--- layout: report year: "2019" month: "05" title: "Reproducible Builds in May 2019" draft: false date: 2019-06-06 12:50:39 --- [![]({{ "/images/reports/2019-05/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) **Welcome to the May 2019 report from the [Reproducible Builds](https://reproducible-builds.org) project!** In our reports we outline the most important things which we have been up to in and around the world of reproducible builds & secure toolchains over the past month. As a quick recap, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users pre-compiled. The motivation behind reproducible builds effort is to ensure no malicious flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing third-parties to come to a consensus on whether a build was compromised. In this month's report, we will cover: * **Media coverage** — *More supply chain attacks, Reproducible Builds at conferences, etc.* * **Upstream news** — *Mozilla updates their add-on policy, etc.* * **Distribution work** — *Debian Installer progress, openSUSE updates.* * **Software development** — *A try.diffoscope.org rewrite, more upstream patches, etc.* * **Misc news** — *From our mailing list, etc.* * **Getting in touch** — *How to contribute, contact details, etc.* If you are interested in contributing to our project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. --- ## Media coverage [![]({{ "/images/reports/2019-05/wired.png#right" | relative_url }})](https://www.wired.com/) * Adam Greenberg reported on [Wired](https://www.wired.com) about the "mysterious hacker group" [Barium](https://www.wired.com/story/barium-supply-chain-hackers/), detailing a single group of malicious actors who appear responsible for a variety supply chain attacks of [CCleaner](https://www.ccleaner.com/), [Asus](https://www.asus.com/) and more, planting backdoors on & gaining access to millions of end-user machines. * The work of Chris Lamb in/around Debian's Reproducible Builds effort [was awarded a Google Open Source Peer Bonus award](https://opensource.googleblog.com/2019/04/google-open-source-peer-bonus-winners.html), a program with the goal of recognising and supporting the ecosystem and sustainability of free software by recognising developers for their contributions to open source projects. * Kushal Das presented at [PyCon](https://us.pycon.org/2019/about/) 2019 on [building reproducible Python applications for secured environments](https://www.youtube.com/watch?v=wRHi8Ui5vWA). Here, Kushal argues that validating the dependencies of the project is as critical as actual project source code, referring to incidents where actors [were able to steal bitcoins using a popular library](https://www.theregister.co.uk/2018/11/26/npm_repo_bitcoin_stealer/). His talk uses the [SecureDrop client application](https://github.com/freedomofpress/securedrop-client) for journalists as an example project to see how to tackle the more general problem. [![]({{ "/images/reports/2019-05/kubecon.png#right" | relative_url }})](https://www.youtube.com/watch?v=X_Sb96EKFPA) * [GitHub](https://github.com/) announced [adding a package registry feature](https://github.com/features/package-registry) which "[suggest but alas not guarantee](https://github.com/ipfs/package-managers/issues/55)" a strong link between the Git repository and the published packages, highlighting the need for Reproducible Builds in this area. * [Andrew Martin](https://www.binarysludge.com/) has [published his slides](https://drive.google.com/a/control-plane.io/file/d/1xUDrcWmB3a_5oMxeIJuqf6vtXZN/view?usp=sharing) for his talk entitled [*Rootless, Reproducible and Hermetic: Secure Container Build Showdown*](https://www.youtube.com/watch?v=X_Sb96EKFPA) that he gave at [KubeCon 2019](https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/). --- ## Upstream news The [IPFS](https://ipfs.io) "[Package Managers Special Interest Group](https://github.com/ipfs/package-managers#readme)" is [gathering research around package management](https://github.com/ipfs/package-managers/blob/master/docs/papers.md), much of which is relevant to the Reproducible Builds effort. [![]({{ "/images/reports/2019-05/buildroot.png#right" | relative_url }})](https://buildroot.org/) Atharva Lele plans to work on reproducible builds for the [Buildroot](https://buildroot.org/) embedded Linux project as part of [Google Summer of Code](https://summerofcode.withgoogle.com/), [ensuring that two instances of buildroot running with the same configuration for the same device yield the same result](https://summerofcode.withgoogle.com/projects/#5992608243908608). [Mozilla](https://www.mozilla.org)'s latest update to the [Firefox](https://www.mozilla.org/en-GB/firefox/) add-on policy [now dictates that add-ons may contain "transpiled, minified or otherwise machine-generated code"](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/AMO/Policy/Reviews-2019-05) but Mozilla needs to review a copy of the human-readable source code. The author must provide this information to Mozilla during submission along with instructions on how to reproduce the build. --- ### Distribution work [![]({{ "/images/reports/2019-05/opensuse.png#right" | relative_url }})](https://www.opensuse.org/) Bernhard M. Wiedemann posted his [monthly Reproducible Builds status update](https://lists.opensuse.org/opensuse-factory/2019-05/msg00341.html) for the [openSUSE](https://opensuse.org/) distribution. Holger Levsen filed a wishlist request requesting that Debian's `.buildinfo` build environment specification documents from the [Debian Long Term Support (LTS)](https://wiki.debian.org/LTS/) project are [also distributed by the build/archive infrastructure](https://bugs.debian.org/929397) so that the reproducibility status of these security packages can be validated. [![]({{ "/images/reports/2019-05/debian.png#left" | relative_url }})](https://debian.org/) There was yet more progress towards making the [Debian Installer](https://www.debian.org/devel/debiah-installer/) images reproducible. Following-on from last months, [Chris Lamb](https://chris-lamb.co.uk/) performed some further testing of the generated images and [requested a status update](https://bugs.debian.org/926242#67) which resulted in a call for testing the [possible removal of a now-obsolete workaround](https://bugs.debian.org/926242#87) that is hindering progress. 68 reviews of Debian packages were added, 30 were updated and 11 were removed this month, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb discovered, identified and triaged two new issue types, the first identifying randomness in [Fontconfig](https://www.freedesktop.org/wiki/Software/fontconfig/) `.uuid` files [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/0b9e9668)] and another [`randomness_in_output_from_perl_deparse`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/430c2d21). Finally, [GNU Guix](https://www.gnu.org/software/guix) announced its [1.0.0 release](https://www.gnu.org/software/guix/blog/2019/gnu-guix-1.0.0-released/). --- ## Software development #### Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream wherever possible. This month, we wrote a large number of such patches, including: [![]({{ "/images/reports/2019-05/notion.png#right" | relative_url }})](https://notionwm.net/) * Arnout Engelen [authored a pull request](https://github.com/raboof/notion/pull/100) to make the binary of the [Notion window manager](https://notionwm.net/) reproducible. * Bernhard M. Wiedemann: * [dvdstyler](https://sourceforge.net/p/dvdstyler/DVDStyler/merge-requests/1/) (`.zip` [ctime](https://en.wikipedia.org/wiki/Ctime)) * [fs-uae](https://build.opensuse.org/request/show/701063) ([already filed upstream](https://github.com/FrodeSolheim/fs-uae/pull/182); zip order, date/`mtime`) * [gettext-runtime](https://build.opensuse.org/request/show/705693) (Use the [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/docs/source-date-epoch/) environment variable) * [gnome-builder](https://build.opensuse.org/request/show/701094) (Drop `environment.pickle` file) * [mrrescue](https://build.opensuse.org/request/show/701742) (`zip -X` modification time) * [mvapich2](https://build.opensuse.org/request/show/705701) (Sort `readdir(2)` call, [already filed upstream](http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2019-April/006837.html)) * [nulloy](https://github.com/nulloy/nulloy/pull/149) (`.zip` timestamps) * [osc](https://github.com/openSUSE/osc/issues/547) (Dependency bug hindering openSUSE reproducible builds) * [pithos](https://build.opensuse.org/request/show/706096) (make `.pyc` files not vary from architecture) * [plata-theme](https://gitlab.com/tista500/plata-theme/merge_requests/3) (zip `mtime`) * [python-Fabric3](https://build.opensuse.org/request/show/702815) (Workaround FTBFS `-j1`) * [python-keystonemiddleware](https://review.opendev.org/657780) (Make tests pass in 2020) * [python-nbconvert](https://bugzilla.opensuse.org/show_bug.cgi?id=1136099) (Fails to build in single-process, `-j1`, mode) * [python-ovirt-engine-sdk](https://gerrit.ovirt.org/100278) (Sort input file list) * [python-requests-toolbelt](https://github.com/requests/toolbelt/issues/270) (Does not build in the year 2021) * [python-rjsmin](https://build.opensuse.org/request/show/703832) (Disable profiling) * [python3-saml](https://github.com/onelogin/python3-saml/pull/140) (Does not build in the year 2020) * [zip](https://build.opensuse.org/request/show/700402) (Add `SOURCE_DATE_EPOCH` clamping of modification times; [also submitted upstream](https://sourceforge.net/p/infozip/patches/25/) and in [distropatches](https://github.com/distropatches/zip/commits/opensuse)) * Chris Lamb: * [#836609](https://bugs.debian.org/836609) re-opened for [nostalgy](https://tracker.debian.org/pkg/nostalgy). * [#928329](https://bugs.debian.org/928329) filed against [fonts-ipaexfont](https://tracker.debian.org/pkg/fonts-ipaexfont). * [#929208](https://bugs.debian.org/929208) filed against [xorg-gtest](https://tracker.debian.org/pkg/xorg-gtest). * [#929609](https://bugs.debian.org/929609) filed against [ndpi](https://tracker.debian.org/pkg/ndpi). * [#929791](https://bugs.debian.org/929791) filed against [ghmm](https://tracker.debian.org/pkg/ghmm). * [#929793](https://bugs.debian.org/929793) filed against [liblopsub](https://tracker.debian.org/pkg/liblopsub). [![]({{ "/images/reports/2019-05/u-boot.png#center" | relative_url }})](https://www.denx.de/wiki/U-Boot/) Finally, Vagrant Cascadian [submitted a patch](https://patchwork.ozlabs.org/patch/1093969/) for [u-boot](https://www.denx.de/wiki/U-Boot/) boot loader fixing reproducibility when building a new type of compressed image. This [was subsequently merged in version `2019.07-rc2`](https://git.denx.de/?p=u-boot.git;a=commit;h=878e2a50b50199cb06ee28df53151e396a29d838). #### diffoscope [![]({{ "/images/reports/2019-05/diffoscope.svg#right" | relative_url }})](https://diffoscope.org) [diffoscope](https://diffoscope.org/) is our in-depth "diff-on-steroids" utility which helps us diagnose reproducibility issues in packages. It does not define reproducibility, but rather provides a helpful and human-readable guidance for packages that are not reproducible, rather than relying essentially-useless diffs. * Chris Lamb: * Support the latest [PyPI](https://pypi.org/) package repository upload requirements by using real [reStructuredText](http://docutils.sourceforge.net/rst.html) comments instead of the `raw` directive [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/edb01aa)] and by stripping out manpage-only parts of the `README` rather than using the `only` directive [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/d06d065)]. * Fix execution of symbolic links that point to the `bin/diffoscope` entry point in a checked-out version of our Git repository by fully resolving the location as part of dynamically calculating Python's module include path. [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/afc63b3)] * Add a [Dockerfile](https://docs.docker.com/engine/reference/builder/) [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/2c37397)] with various subsequent fixups [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/2cb8206)][[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/8f67fcb)][[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/ba4b2ae)]. * Published the resulting Docker image in [diffoscope's container registry](https://salsa.debian.org/reproducible-builds/diffoscope/container_registry) and updated the [diffoscope homepage](https://diffoscope.org/) to provide "quick start" instructions on how to use diffoscope via this image. * Mattia Rizzolo: * Uploaded version `115` [to Debian experimental](https://tracker.debian.org/news/1040177/accepted-diffoscope-115-source-all-into-experimental/). * Adjust various build and test-dependencies, including specifying the [ffmpeg](https://ffmpeg.org/) video encoding tool/library and the [Black](https://ffmpeg.org/) code formatter [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/0eddfab)] in the build-dependencies [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/d2d3dec)] and reinstating the [oggvideotools](https://sourceforge.net/projects/oggvideotools/) and `procyon-decompiler` as test dependencies, now that are no-longer buggy [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6940757)], etc. * Make the Debian autopkgtests not fail when a limited subset of "required tools" are temporarily unavailable. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f584fa2)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/3d74240)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2e11182)] In addition, Santiago Torres altered the behaviour of the tests to ensure compatibility with various versions of [`file(1)`]() [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/0f02296)] and Vagrant Cascadian added support for various external tools in [GNU Guix](https://www.gnu.org/software/guix/) [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/7f3416f)] and updated the version of *diffoscope* in that distribution [[...](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ff793da66918ace85048f90dc069415ef067ba06)]. #### try.diffoscope.org [![]({{ "/images/reports/2019-05/trydiffoscope.png#right" | relative_url }})](https://try.diffoscope.org/) Chris Lamb made a large number of following changes to the web-based ("no installation required") version of the *[diffoscope](https://diffoscope.org)* tool, [try.diffoscope.org](https://try.diffoscope.org): * Ported the entire site to Python 3 and [Django](https://www.djangoproject.com/) 2.x as [Python 2.x is due for deprecation](https://pythonclock.org/). This required updates to a huge number of parts around the site including but not limited to completely reconfiguring and integrating the [Celery](http://www.celeryproject.org/) queue processor, all the string formatting, etc. * Moved to using the published/public [Docker](https://www.docker.com/) image to execute builds instead rolling our own container. * Updated and upgraded the underlying operating system to the Debian *stable* distribution. * Moved the [canonical Git repository](https://salsa.debian.org/reproducible-builds/try.diffoscope.org) from GitHub to the [Reproducible Builds group on salsa.debian.org](https://salsa.debian.org/reproducible-builds/try.diffoscope.org), requiring moving to GitLab's own [continuous integration (CI) support](https://docs.gitlab.com/ee/ci/) from [Travis CI](https://travis-ci.org/), working around the aggressive firewall (exclusively outgoing ports 80/443) applied to the [Salsa](https://salsa.debian.org/)-based CI runners. * Avoid having to update the [Let's Encrypt](https://letsencrypt.org/)-provided SSL certificate manually every 90 days by moving to using [Certbot](https://certbot.eff.org/about/) in `--auto` mode. #### Test framework We operate a comprehensive [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org). The following changes were done in the last month: * Holger Levsen made the following ([Debian](https://www.debian.org/)-related changes): * Reduce the number of `cron(8)` mails for synchronising `.buildinfo` files from eight to one per day. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/40571647)] * Run `rsync2buildinfos.debian.net` script every other hour now that it just produces one mail per day. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/67c819fb)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9db63b7b)] * Execute the package scheduler every 2 hours (instead of 3). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/bb9b49f2)] * Switch the [Codethink](https://www.codethink.co.uk/) and OSUOSL nodes to use our updated email relay system. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/45532738)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/88bd5987)] * Deal with the (rare) cases of `.buildinfo` files with the same name. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c7d6f107)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/951dd1c0)] * Save and mail the package scheduler results once a day instead of mailing ~8 times a day. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5e3f7d34)] * In addition, Holger Levsen made the following distribution-agnositic changes: * Notify the `#reproducible-builds` (not `#debian-reproducible`) about Jenkins rebooting and send notifications about offline hosts to this former channel. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a51ffc69)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c9d322ef)] * Prevent the Jenkins log from growing to over 100G in size. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8f21110b)] * Mattia Rizzolo: * Use a special code so that remote builds can abort themselves by passing back the command to the "master". [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/17303def)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/33ea96a3)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/923d22a2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8f807691)] * Fix a pattern matching bug to ensure all "zombie" processes are found. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/33fa12bb)] * [flake8](http://flake8.pycqa.org/en/latest/) the `chroot-installation.yaml.py` file. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/839af096)] * Set a known [HTTP User Agent](https://en.wikipedia.org/wiki/User_agent#Use_in_HTTP) for Git, so that server can recognise us. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c93bdaeb)] * Allow network access for the [`debian-installer-netboot-images`](https://tracker.debian.org/pkg/debian-installer-netboot-images) Debian package. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f1e641b8)] Finally, Vagrant Cascadian removed the deprecated `--buildinfo-id` from the `pbuilder(8)` configuration. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6376dd4a)] and Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/08b024a2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/63cff4ef)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/941f6fd1)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/db0e4ecc)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5f59a8f8)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2401e4ee)] Mattia Rizzolo [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/dd86f22e)] and Vagrant Cascadian all performed a large amount of build node maintenance, system & Jenkins administration. #### Project website [![]({{ "/images/reports/2019-05/website.png#right" | relative_url }})](https://reproducible-builds.org/) Chris Lamb added various fixes for larger/smaller screens [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/edef6f8)], added a logo suitable for printing physical pin badges [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/d78fd45)] and refreshed the opening copy text on our [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/docs/source-date-epoch) page. Bernhard M. Wiedemann then [documented a more concise C code example](https://reproducible-builds.org/docs/source-date-epoch/#c) for parsing the `SOURCE_DATE_EPOCH` environment variable [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/547732f)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/e1efd6f)] and Holger Levsen added a [link to a specific bug](https://bugzilla.opensuse.org/show_bug.cgi?id=1133809) blocking progress in [openSUSE](https://opensuse.org/) to our [*Who is involved?*]({{ "/who/" | relative_url }}) page [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d9f4bce)]. --- ## Misc news * On [our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/) this month [Lars Wirzenius](https://liw.fi/) asked [various questions about reproducible builds](https://lists.reproducible-builds.org/pipermail/rb-general/2019-May/001566.html) and their bearing on building a distributed continuous integration system which received many replies ([view thread index](https://lists.reproducible-builds.org/pipermail/rb-general/2019-May/thread.html#1566)). [![]({{ "/images/reports/2019-05/profitbricks.png#right" | relative_url }})](https://www.profitbricks.com) * The server powering [`lists.reproducible-builds.org`](http://lists.reproducible-builds.org/) changed home. Thanks to [`potager.org`](https://potager.org/) for hosting us all this time and many thanks to [Profitbricks](https://www.profitbricks.com) for hosting our new mail server as well as all the other nodes over the years. * Mo Zhou wrote a [detailed policy for deep learning software](https://salsa.debian.org/lumin/deeplearning-policy) for the [Debian](https://debian.org) distribution which touches on the reproducibility of data models. Lastly, Sam Hartman, the current [Debian Project Leader](https://www.debian.org/devel/leader), wrote on the [`debian-devel`](https://lists.debian.org/debian-devel) mailing list: > The reproducible builds world has gotten a lot further with bit-for-bit identical builds than I ever imagined they would. [[...](https://lists.debian.org/debian-devel/2019/05/msg00355.html)] Thanks, Sam! --- ## Getting in touch ![]({{ "/images/reports/2019-05/irc.png#right" | relative_url }}) If you are interested in contributing the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * IRC: `#reproducible-builds` on `irc.oftc.net`. * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds) * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)
--- This month's report was written by Arnout Engelen, Bernhard M. Wiedemann, [Chris Lamb](https://chris-lamb.co.uk/), [Holger Levsen](http://layer-acht.org/thinking/), Mattia Rizzolo and Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.