--- layout: report year: "2019" month: "04" title: "Reproducible Builds in April 2019" draft: false date: 2019-05-05 17:08:27 --- **Welcome to the April 2019 report from the [Reproducible Builds](https://reproducible-builds.org) project!** In these now-monthly reports we will outline the most important things which we have been up to in and around the world of reproducible builds & secure toolchains. [![]({{ "/images/reports/2019-04/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) As a quick recap, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users pre-compiled. The motivation behind reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. In this month's report, we will cover: * **Media coverage** — *Compromised toolchains, what makes a good digital product?, etc.* * **Upstream news** — *Scala and Go working on reproducibility, etc.* * **Distribution work** — *Distributing build certificates, an update from openSUSE, etc.* * **Software development** — *New features in diffoscope, yet more test framework development, etc* * **Misc news** — *From our mailing list, etc.* * **Getting in touch** — *How to contribute, etc* --- ## Media coverage * The [SecureList](https://securelist.com) website [reported on Operation "ShadowHammer"](https://securelist.com/operation-shadowhammer-a-high-profile-supply-chain-attack/90380/), a high-profile supply chain attack involving the [ASUS](https://en.wikipedia.org/wiki/Asus) Live Update Utility. As their post describes in more detail tampering with binaries would usually break the digital signature, but in this case the digital signature itself appeared to have been compromised. ([Read more](https://securelist.com/operation-shadowhammer-a-high-profile-supply-chain-attack/90380/)) [![]({{ "/images/reports/2019-04/lwn.png#right" | relative_url }})](https://lwn.net/) * [Linux Weekly News (LWN)](https://lwn.net/) covered the [recent `bootstrap-sass` backdoor incident](https://lwn.net/Articles/785386/) which speaks to the impact and potential prevalence of supply-chain and mirror-based attacks. [David A. Wheeler](https://dwheeler.com) also [published an essay on the incident](https://dwheeler.com/essays/bootstrap-sass-subversion.html) that explicitly proposes reproducible builds as a potential way to reduce the impact of such attacks in the future. * There was an interesting discussion on [Hacker News](https://news.ycombinator.com/) regarding the release of [WAPM](https://wapm.io/), a package manager for [WebAssembly](https://webassembly.org/) packages (typically embedded into browsers and web-pages). In [the discussion there was a query](https://news.ycombinator.com/item?id=19732794) and distinction raised by commenter *whyrusleeping* between the ability to reproduce any generated packages versus simply signing packages in the usual manner which received warm reception by the upstream authors. * An issue was reported against the [`libsodium`](https://libsodium.org) crypto library which [asked for clarification why the `1.0.17` release was modified on the download server](https://github.com/jedisct1/libsodium/issues/813). In response to this, [a pull request was created by Philip Crockett](https://github.com/sodiumoxide/sodiumoxide/pull/329) to verify the project with the `minisign` algorithm instead of `sha2`. * Bobby Richter [proposed the addition of reproducible builds](https://github.com/TheDigitalStandard/TheDigitalStandard/pull/115) as indicator of good digital products. * [Anmol Sarma](https://www.anmolsarma.in/) wrote a blog post requesting that developers "[Stop Memsettings Structures](https://www.anmolsarma.in/post/stop-struct-memset/)". This is relevant to the Project as `memset(3)` system call is often used to ensure deterministic output of packages or of binaries themselves; if the build artifacts contain the contents uninitialised memory, to ensure a reproducible build a developer would typically "zero out" the memory using `memset(3)` to ensure that it does not contain the so-called random data. ## Upstream news [![]({{ "/images/reports/2019-04/scala.png#right" | relative_url }})](https://www.scala-lang.org/) The first non-trivial library written in the [Scala](https://www.scala-lang.org/) programming language on the [Java Virtual Machine](https://reproducible-builds.org/docs/jvm/) was released with Arnout Engelen's [`sbt-reproducible-builds`](https://github.com/raboof/sbt-reproducible-builds) plugin enabled during the build. This resulted in [Akka 2.5.22](https://akka.io/blog/news/2019/04/03/akka-2.5.22-released) becoming reproducible, both for the artifacts built with version [2.12.8](https://arnout.engelen.eu/rb/akka/2.12/2.5.22/) and 2.13.0-RC1 of the Scala compiler. For 2.12.8, the original release was performed on a Mac and the validation was done on a Debian-based machine, so it appears the build is reproducible across diverse systems. ([Mailing list thread](https://lists.reproducible-builds.org/pipermail/rb-general/2019-April/001529.html)) Jeremiah "DTMB" Orians announced the 1.3.0 release of [M2-Planet](https://github.com/oriansj/M2-Planet), a self-hosting C compiler written in a subset of the features it supports. It has been bootstrapped entirely from hexadecimal (!) with 100% reproducible output/binaries. This new release sports a self-hosting port for an additional architecture amongst other changes. Being "self-hosted" is an important property as it can provide a method of validating the legitimancy of the build toolchain. [![]({{ "/images/reports/2019-04/golang.png#left" | relative_url }})](https://golang.org/) The [Go](https://golang.org) programming language has been making progress in making their builds reproducible. In 2016, Ximin Luo had created [issue #16860](https://github.com/golang/go/issues/16860) requesting that the compiler generates the same result regardless of the path in which the package is built. However, progress was recently made in [Change #173344](https://go-review.googlesource.com/c/go/+/173344/) (and adjacent) that will permit a `-trimpath` mode that will generate binaries that do not contain any local path names, similar to [`-fpath-prefix-map`](https://reproducible-builds.org/specs/build-path-prefix-map/). The [fontconfig](https://www.freedesktop.org/wiki/Software/fontconfig/) library for configuring and customising font access in a number of distributions [announced they had merged patches](https://lists.freedesktop.org/archives/fontconfig/2019-April/006508.html) to allow various cache files to be reproducible. This is after Chris Lamb posted a historical summary and [a request for action](https://lists.freedesktop.org/archives/fontconfig/2019-January/006420.html) to [Fontconfig](https://www.freedesktop.org/wiki/Software/fontconfig/)'s mailing list in January 2019 ### Distribution work [![]({{ "/images/reports/2019-04/debian.png#right" | relative_url }})](https://debian.org/) In Debian, [Chris Lamb](https://chris-lamb.co.uk/) added 90 reviews of Debian packages, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html) and 14 issues were automatically removed. Chris also added two issue types: [`build_date_in_egg_info_directory_name`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/61bf6116) & [`randomness_in_perl6_precompiled_libraries`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/b718ab29). Holger Levsen started a [discussion regarding the distribution of `.buildinfo` files](https://lists.reproducible-builds.org/pipermail/rb-general/2019-April/001550.html). These files record the environment that was used as part of a particular build in order that — along with the source code — ensure that the aforementioned environment can be recreated at a later date to reproduce the exact binary. Distributing these files is important so that others can validate that a build is actually reproducible. In his post, Holger refers to two services that now exist, [buildinfo.debian.net](https://buildinfo.debian.net) and [buildinfos.debian.net](https://buildinfos.debian.net). In addition, Holger restarted a long-running discussion regarding the [reproducibility status of Debian buster](https://lists.reproducible-builds.org/pipermail/rb-general/2019-April/001549.html) touching on questions of potentially performing mass rebuilds of all packages in order that they use updated toolchains. There was yet more progress towards making the [Debian Installer](https://www.debian.org/devel/debiah-installer/) images reproducible. Following-on from last months, [Chris Lamb](https://chris-lamb.co.uk/) performed some further testing of the generated images. Cyril Brulebois then made an [upload of the `debian-installer` package](https://bugs.debian.org/920676#50) to Debian that included a number of Chris' patches and Vagrant Cascadian filed a patch to [fix the reproducibility of "u-boot" images](https://salsa.debian.org/installer-team/debian-installer/commit/deeee34bc0ee5ec879182111b809896752ad0df9) by using `-n` argument to `gzip(1)`. [![]({{ "/images/reports/2019-04/opensuse.png#left" | relative_url }})](https://www.opensuse.org/) Bernhard M. Wiedemann posted his [monthly Reproducible Builds status update](https://lists.opensuse.org/opensuse-factory/2019-04/msg00414.html) for the [openSUSE](https://opensuse.org/) distribution. Bernhard also posted to our mailing list regarding [enabling the normalisation of file modification times in Python `.pyc` files](https://lists.reproducible-builds.org/pipermail/rb-general/2019-April/001548.html) and opened issue [#1133809](https://bugzilla.opensuse.org/show_bug.cgi?id=1133809) in the openSUSE bug tracker.
## Software development #### Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Bernhard M. Wiedemann: * [`dovecot23`](https://bugzilla.opensuse.org/show_bug.cgi?id=1131699) (Report random build failure with -j1) * [`rash`](https://github.com/willghatch/racket-rash/issues/52) (Report parallelism-related nondeterminism) * [`maven-compiler-plugin`](https://issues.apache.org/jira/browse/MCOMPILER-380) (Report copyright year / date) * [`python-textX`](https://github.com/textX/textX/pull/181) (Sort Python [glob](https://docs.python.org/3/library/glob.html)) * [`python-pyOpenSSL`](https://github.com/pyca/pyopenssl/pull/828) (Fix FTBFS-2020) * [`python-pyngus`](https://github.com/kgiusti/pyngus/pull/13) (Fix FTBFS-2028) * [`python-ldap`](https://github.com/python-ldap/python-ldap/pull/277) (Fix FTBFS-2027) * [`python-pysaml2`](https://github.com/IdentityPython/pysaml2/pull/606) (Fix FTBFS-2024) * [`gnutls`](https://gitlab.com/gnutls/gnutls/merge_requests/979) (Fix FTBFS-2024) * [`python-autobahn`](https://build.opensuse.org/request/show/692210) (CPU-detection - consider upstreaming some variant) * [`python-pocketsphinx-python`](https://build.opensuse.org/request/show/692544) ([upstream](https://github.com/bambocher/pocketsphinx-python/pull/45)) (sort Python `glob` in `setup.py`) * [`python-py-ubjson`](https://build.opensuse.org/request/show/692636) ([upstream](https://github.com/Iotic-Labs/py-ubjson/pull/7)) (sort Python `glob` in `setup.py`) * [`python-debtcollector/python-openstackdocstheme`](https://review.openstack.org/652669) (Python date) * [`branding-openSUSE`](https://github.com/openSUSE/branding/pull/111) (Rediscovered already-fixed parallelism race) * [`python-irc`](https://build.opensuse.org/request/show/699679) (Drop file with varying `.pyc` timestamp) * Chris Lamb: * [#926298](https://bugs.debian.org/926298) filed against [`adms`](https://tracker.debian.org/pkg/adms) ([merged upstream](https://github.com/Qucs/ADMS/pull/84#issuecomment-484791782)). * [#926300](https://bugs.debian.org/926300) filed against [`qpid-proton`](https://tracker.debian.org/pkg/qpid-proton). * [#926301](https://bugs.debian.org/926301) filed against [`coda`](https://tracker.debian.org/pkg/coda). * [#926421](https://bugs.debian.org/926421) filed against [`netcdf-parallel`](https://tracker.debian.org/pkg/netcdf-parallel). * [#928183](https://bugs.debian.org/928183) filed against [`fim`](https://tracker.debian.org/pkg/fim). * In addition, Chris' previous patch to the `shadow` password utility [was merged upstream](https://github.com/shadow-maint/shadow/pull/146#issuecomment-485286090). * Vagrant Cascadian: * [linux](https://salsa.debian.org/kernel-team/linux/merge_requests/140): Sort list of modules before adding to `.json` file. [[...](https://salsa.debian.org/kernel-team/linux/commit/58ef63e9e2c71ffd8a21e9c620db71cb96d2d5a9)] #### diffoscope [![]({{ "/images/reports/2019-04/diffoscope.svg#right" | relative_url }})](https://diffoscope.org) [diffoscope](https://diffoscope.org/) is our in-depth "diff-on-steroids" utility which helps us diagnose reproducibility issues in packages. It does not define reproducibility, but rather provides a helpful and human-readable guidance for packages that are not reproducible, rather than relying essentially-useless diffs. This month, Chris Lamb did a lot of development of diffoscope, including: * [Updating the certificate](https://github.com/lamby/try.diffoscope.org/commit/aa3cc35451dd7fedfdc30af7b248b39d0e9f7898) of the [try.diffoscope.org](https://try.diffoscope.org) web-based version of the tool. * [Uploaded version 114](https://tracker.debian.org/news/1038940/accepted-diffoscope-114-source-all-into-experimental/) to the Debian *experimental* distribution and made the corresponding upload to the [PyPI](https://pypi.org/project/diffoscope/) package repository. * Added support for semantic comparison of [GnuPG](https://www.gnupg.org/) "keybox" (`.kbx`) files. ([#871244](https://bugs.debian.org/871244)) * Add the ability to treat missing tools as failures if a "magic" environment variable is detected in order to facilitate interpreting required tools on the [Debian autopkgtests](https://ci.debian.net/) as actual test failures, rather than skipping them. The behaviour of the existing testsuite remains unchanged. ([#905885](https://bugs.debian.org/905885)) * Filed a "request for packaging" for the [annocheck](https://sourceware.org/git/?p=annobin.git) tool which can be used to "analyse an application's compilation". This is as part of [an outstanding wishlist issue](https://salsa.debian.org/reproducible-builds/diffoscope/issues/51). ([#926470](https://bugs.debian.org/926470)) * Consolidated on a single alias as the exception value across the entire codebase. [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/39e7b6b)] In addition, Vibhu Agrawal ensured that diffoscope failed more gracefully when running out of diskspace to resolve Debian bug [#874582](https://bugs.debian.org/874582) and Vagrant Cascadian [updated to diffoscope 114](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c31bec88a818a0ee112e1e80222e7beffa463057) in [GNU Guix](https://www.gnu.org/software/guix/). Thanks! #### strip-nondeterminism [strip-nondeterminism](https://tracker.debian.org/pkg/strip-nondeterminism) is our tool to remove specific non-deterministic results from a completed build. It is used automatically in most Debian package builds. This month, Chris Lamb made the following improvements: * Workaround [`Archive::Zip`](https://metacpan.org/pod/Archive::Zip)'s incorrect handling of the `localExtraField` class member field by monkey-patching the accessor methods to always return normalised values. This fixes the normalisation of Unix ownership metadata within `.zip` and `.epub` files. ([#858431](https://bugs.debian.org/858431)) * Actually check the return status from `Archive::Zip` when writing file to disk. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism.git/commit/b164403)] * Catch an edge-case where we can't parse the length of a particular field within `.zip` files. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism.git/commit/783d44e)] Chris then [uploaded version `1.1.3-1`](https://tracker.debian.org/news/1038943/accepted-strip-nondeterminism-113-1-source-all-into-experimental/) to the Debian *experimental* distribution. #### Project website Chris Lamb made a number of improvements to [our project website](https://reproducible-builds.org) this month, including: [![]({{ "/images/reports/2019-04/website.png#right" | relative_url }})](https://reproducible-builds.org/) * Using an explicit "draft" boolean flag for posts. [Jekyll](https://jekyllrb.com/) in Debian stable silently (!) does not support the `where_exp` filter. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/2f27517)] * Moving more pages away from the old design with HTML to [Markdown](https://en.wikipedia.org/wiki/Markdown) formatting and the new design template. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/19c7951)] * Adding a simple [Makefile](https://en.wikipedia.org/wiki/Makefile) to implicitly document how to build the site [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/172bcfd)] and add a simple `.gitlab-ci.yml` to test branches/builds [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/595d29b)]. * Adding as simple "lint" command so we can see how many pages are using the old style. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/2b12ff7)] * Adding an explicit link to our "[Who is involved?](https://reproducible-builds.org/who/)" page in the footer of the newer design [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/2d14946)] and add a link to donation page [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/7a95a81)]. * Moved various bits of infrastructure to support a monthly report structure. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/eb0ae67)] #### Test framework We operate a comprehensive [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org). The following changes were done in the last month: * Holger Levsen ([Debian](https://www.debian.org/)-related changes): * Add new experimental [buildinfos.debian.net](https://buildinfos.debian.net) service. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4bcef9ec)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1012c656)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/41152533)] * Allow pushing of `.buildinfo` files from [coccia](https://db.debian.org/machines.cgi?host=coccia). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/cce3dbcd)] * Permit [rsync](https://rsync.samba.org/) to write into subdirectories. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/002ca47d)] * Include the meta "pool" job in the overall job health view. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3aabae72)] * Add support for host-specific SSH `authorized_keys` files used on a particular build node. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c5b38ef7)] * Show link to maintenance jobs for offline nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/416c60f4)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2639e5ef)] * Increase the job timeout for some runners from 3 to 5 days. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/293b6563)] * Don't try to turn Jenkins or nodes offline too quickly. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/413d585a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0e33802d)] * Fix pbuilder lock files if necessary. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/dd632698)] * Mattia Rizzolo: * Special-case the `debian-installer` package when building to allow it access to the internet.. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e3117ca2)] * Force installing the `debootstrap` from [`stretch` backports](https://backports.debian.org/) and remove `cdebootstrap`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9e252095)] * Install the `python3-yaml` package on nodes as it is needed by the deploy script. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/46d33b21)] * Add/update the new `reproducible-builds.org` [MX records](https://en.wikipedia.org/wiki/MX_record). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9ddd1042)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/711267ec)] * Fix typo in comment; thanks to `ijc` for reporting! [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2435823c)] Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4a79527a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a24c3aa9)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/363a02f3)], Mattia Rizzolo [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9d4d39d1)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a6412217)] all performed a large amount of build node maintenance, system & Jenkins administration and Chris Lamb provided a patch to avoid double spaces in IRC notifications [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f4b80011)].
## Misc news * AJ Jordan updated `reprotest`, our "end-user" tool to build arbitrary software and check it for reproducibility — to reference `--no-clean-on-error` in the `--store-dir text`. [[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/19b0669)] * David A. Wheeler started a thread on our mailing list regarding [changing the front page definition](https://lists.reproducible-builds.org/pipermail/rb-general/2019-April/001523.html) and, Daniel Shahaf [posted an April Fool's joke](https://lists.reproducible-builds.org/pipermail/rb-general/2019-April/001517.html). * Whilst the Reproducible Builds project intended to participate in [Google Summer of Code](https://summerofcode.withgoogle.com/) and [Outreachy](https://www.outreachy.org/) in 2019 we sadly did not find any suitable students. We do plan to be involved in future rounds wherever possible. * Chris Lamb noticed that the SUSv3/POSIX UNIX specification mentions that for portability-reasons [the character string that identifies the timezone description should begin with a colon character](https://unix.stackexchange.com/a/48104/222284) which may have future implications regarding ensuring a particular timezone to ensure a reproducible build. --- ## Getting in touch If you are interested in contributing the Reproducible Builds project, please visit our [Contribute](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * Mailing list: [`rb-general`](https://lists.reproducible-builds.org/listinfo/rb-general) * IRC: `#reproducible-builds` on `irc.oftc.net`. * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)
--- This month's report was written by Arnout Engelen, Bernhard M. Wiedemann, [Chris Lamb](https://chris-lamb.co.uk/), [Holger Levsen](http://layer-acht.org/thinking/), Mattia Rizzolo and Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.