--- layout: report year: "2021" month: "03" title: "Reproducible Builds in March 2021" draft: false date: 2021-04-07 16:26:15 --- [![]({{ "/images/reports/2021-03/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) **Welcome to the March 2021 report from the [Reproducible Builds](https://reproducible-builds.org) project!** In our monthly reports, we try to outline the most important things that have happened in the reproducible builds community. If you are interested in contributing to the project, though, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on [our website]({{ "/" | relative_url }}).
[![]({{ "/images/reports/2021-03/fdroid.png#right" | relative_url }})](https://www.f-droid.org) [F-Droid](https://www.f-droid.org/) is a large repository of open source applications for the Google Android platform. This month, Felix C. Stegerman announced [*apksigcopier*](https://github.com/obfusk/apksigcopier), a new tool for copying signatures for `.apk` files from a signed `.apk` file to an unsigned one which is necessary in order to verify reproducibly of F-Droid components. Felix filed an [Intent to Package (ITP)](https://wiki.debian.org/ITP) bug in Debian to include it in that distribution as well ([#986179](https://bugs.debian.org/986179)). On 9th March, the Linux Foundation [announced](https://linuxfoundation.org/en/press-release/linux-foundation-announces-free-sigstore-signing-service-to-confirm-origin-and-authenticity-of-software/) the [*sigstore*](https://sigstore.dev/what_is_sigstore/#what-is-sigstore) project, which is a centralised service that allows developers to cryptographically sign and store signatures for release artifacts. *sigstore* attempts to help developers who don't wish to manage their own signing keypairs. [![]({{ "/images/reports/2021-03/openssf.png#right" | relative_url }})](https://openssf.org/) A [discussion was started on Hacker News this month](https://news.ycombinator.com/item?id=26602033) regarding [OpenSSF](https://openssf.org/), a broad technical initiative aiming to focus on vulnerability disclosures, security tooling as well other related threats to open source projects. At the time of writing, the HN discussion has over 70 comments, including input from members of OpenSSF itself. On [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/), Felix C. Stegerman followed-up to a thread in January 2021 regarding [reproducible Python `.pyc` files](https://lists.reproducible-builds.org/pipermail/rb-general/2021-March/002207.html). In addition, Jan Nieuwenhuizen [announced the release of GNU Mes version 0.23](https://lists.reproducible-builds.org/pipermail/rb-general/2021-March/002209.html). [Mes](https://www.gnu.org/software/mes/), a Scheme interpreter and C compiler designed for bootstrapping a base GNU system, was ported to the ARM architecture and can now be used in the [GNU Guix "Reduced Binary Seed" bootstrap](https://guix.gnu.org/blog/2020/guix-further-reduces-bootstrap-seed-to-25). Elsewhere in supply-chain security news, it was discovered that hackers added backdoors to the source code for the PHP programming language after breaching an internal Git server. The malicious code would have made websites vulnerable to a complete takeover including stealing credit card and other sensitive personal information. ([ArsTechnica story](https://arstechnica.com/gadgets/2021/03/hackers-backdoor-php-source-code-after-breaching-internal-git-server/)).
## Software development ### Distribution work [![]({{ "/images/reports/2021-03/coreboot.png#right" | relative_url }})](https://www.coreboot.org/) [Coreboot](https://www.coreboot.org/) is a project that provides a fast, secure and free software alternative boot experience for modern computers and embedded systems. This month, Alexander "*lynxis*" Couzens worked on improving support for Coreboot's payloads to be reproducible. Whilst Coreboot itself is reproducible, not all of its firmware payloads are. However, *lynxis*'s new patches now pass build environment variables (e.g. `TZ`, [`SOURCE_DATE_EPOCH`]({{ "/specs/source-date-epoch/" | relative_url }}), `LANG`, etc.) to the build systems of the respective payloads. [[...](https://review.coreboot.org/q/topic:%22reproducible%22+(status:open%20OR%20status:merged))]
[![]({{ "/images/reports/2021-03/debian.png#right" | relative_url }})](https://debian.org/) When building Debian packages, `dpkg` currently passes options to the underlying build system to strip out the build path from generated binaries. However, many binaries still end up including the build path because they embed the entire compiler command-line which includes, ironically, the very flags that specify the build path to facilitate stripping it out. Vagrant Cascadian therefore [filed a bug against the Debian `dpkg` package](https://bugs.debian.org/985553) to use [GCC](https://gcc.gnu.org/)'s `.spec` files to specify the `fixfilepath` and `fixdebugpath` options. This supplies the build path to GCC via the `DEB_BUILD_PATH` environment variable, thus avoid passing the path on the command-line itself. Related to this, it was noticed that [Debian unstable reached 85% reproducibility](https://tests.reproducible-builds.org/debian/unstable/index_suite_amd64_stats.html) for the first time since enabling variations in the build path. Frédéric Pierret has been working on a partial copy of the [`snapshot.debian.org`](https://snapshot.debian.org/) "wayback machine" service, limited to the packages needed to rebuild Debian *bullseye* on the `amd64` architecture. This is to workaround some limitations of `snapshot.debian.org`. Whilst the mirror itself is reachable at [`debian.notset.fr`](https://debian.notset.fr/snapshot/), the software to creating it is [available in Frédéric's Git repository](https://github.com/fepitre/snapshot-mirror). Currently, Frédéric's service has mirrored 4 months of the archive over two weeks, but needs approximately 3-5 years of content in order to fully rebuild *bullseye*. To that end, a [request was made to the Debian system administrators](https://rt.debian.org/Ticket/Display.html?id=8547) to obtain better access to `snapshot.debian.org` to accelerate the initial seeding. 53 reviews of Debian packages were added, 25 were updated and 22 were removed this month adding to our [extensive knowledge of identified issues](https://tests.reproducible-builds.org/debian/index_issues.html).
[![]({{ "/images/reports/2021-03/opensuse.png#right" | relative_url }})](https://www.opensuse.org/) Bernhard M. Wiedemann posted his [monthly reproducible builds status report](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/6H4AO7BGHXTGUUGWKLDB5VXAZEEIV6SG/) for the [openSUSE](https://www.opensuse.org/) distribution.
[![]({{ "/images/reports/2021-03/nixos.png#right" | relative_url }})](https://nixos.org/) [NixOS](https://nixos.org) continues to inch towards their milestone of having a fully-reproducible minimal installation ISO: the work by Frederik Rietdijk to make the Python packages reproducible [has been merged](https://github.com/NixOS/nixpkgs/pull/107965) and the PR to [build GCC reproducibly](https://github.com/NixOS/nixpkgs/pull/112928) is also progressing. In the mean time a problem with [ruby was found and fixed](https://github.com/nixos/nixpkgs/pull/118551) by Tom Berek and a fix for a [problem with the gi-docgen generation of Pango documentation](https://github.com/NixOS/nixpkgs/issues/118910) is in progress.
### [*diffoscope*](https://diffoscope.org) [![]({{ "/images/reports/2021-03/diffoscope.svg#right" | relative_url }})](https://diffoscope.org) [*diffoscope*](https://diffoscope.org) is the Reproducible Build's project in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it provides human-readable diffs from many kinds of binary formats. This month, [Chris Lamb](https://chris-lamb.co.uk) made a large number of changes (including releasing [version 169](https://diffoscope.org/news/diffoscope-169-released/) [version 170](https://diffoscope.org/news/diffoscope-170-released/) and [version 171](https://diffoscope.org/news/diffoscope-171-released/): * New features: * If `zipinfo(1)` shows a difference but we cannot uncover a difference within the underlying `.zip` or `.apk` file, add a comment to the output and actually show the binary comparison. ([#246](https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/246)) * Ensure all our temporary directories have useful names. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/e5fc1c4)] * Ignore `--debug` and similar arguments when creating a (hopefully-useful) temporary directory suffix. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/fc310cf)] * Optimisations: * Avoid frequent long lines in RPM header outputs that cause extremely slow HTML output generation. ([#245](https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/245)) * Use larger read buffer block sizes when extracting files from archives. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f25c0f6)] * Use a much-shorter HTML class name instead of `diffponct` to optimise HTML output. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/9d9520b)] * Output improvements: * Make `error extracting X, falling back to binary comparison 'Y'` error message in *diffoscope*'s output nicer. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/8bdc89a)] * Don't emit "Unable to stat file" debug messages at all. We have entirely-artificial directory "entries" such as ELF sections which, of course, will never exist as files. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/0d5f7cb)] * Logging improvements: * Add the target directory when logging which directory we are extracting containers to. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/73cf490)] * Format report size messages when generating HTML reports. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/ee963ce)] * Don't emit a `Returning a FooContainer` logging message too, as we already emit `Instantiating a FooContainer` log message. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/25634e3)] * Reduce "Unable to stat file" warnings to debug messages as these are sometimes by design. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/fa16873)] * Misc improvements: * Clarify a comment regarding not extracting excluded files. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/7e31600)] * Remove trailing newline from updated test file (re: [#243](https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/243)). [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/27e2743)] * Fix `test_libmix_differences` failure on openSUSE *Tumbleweed*. ([#244](https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/244)) * Move `test_rpm` to use the `assert_diff` utility helper. In addition Hans-Christoph Steiner added a `diffoscope.tools.get_tools` method to support programmatically fetch *diffoscope*'s internal config [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/248b06a)], Mattia Rizzolo updated the tests to not require a tool when it *wasn't* required as well as to correct a misleading reason for skipping, Roland Clobus made *diffoscope* more tolerant of malformed Debian `.changes`. files [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/acd8c55)] and Vagrant Cascadian updated a test so that it would not be run if a required too was not available [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/64d91ab)]. ### Website and documentation Several changes were made to the [main Reproducible Builds website and documentation](https://reproducible-builds.org/) this month. Arnout Engelen, for example, updated the configuration to avoid a conflict between `jekyll-polyglot` and `sass` [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/18b19a7)] as well as replacing an outdated [NixOS](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7313f07)-related link to a pull request [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7313f07)]. In addition, Chris Lamb fixed some links in old reports [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/35bd350)], Frédéric Pierre updated the entry for [QubesOS](https://www.qubes-os.org/) on our [list of partner projects]({{ "/who/" | relative_url }}), adding an external tests page [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1481bfb)], and Vagrant Cascadian added a 'light' variant of the Reproducible Builds logo [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ada2429)]. #### Upstream patches * Bernhard M. Wiedemann: * [`kio/extra-cmake-modules`](https://invent.kde.org/frameworks/extra-cmake-modules/-/merge_requests/101) (toolchain, normalise `.tar` access times) * [`kismet`](https://build.opensuse.org/request/show/875888) (fix compilation with date patch, needs upstreaming) * [`libkrunfw`](https://build.opensuse.org/request/show/877718) (user, host and date variations) * Chris Lamb: * [#885326](https://bugs.debian.org/885326) filed against [`flask-peewee`](https://tracker.debian.org/pkg/flask-peewee). * [#977487](https://bugs.debian.org/977487) filed against [`pyvows`](https://tracker.debian.org/pkg/pyvows). * [#983852](https://bugs.debian.org/983852) filed against [`python-scrapy`](https://tracker.debian.org/pkg/python-scrapy) ([forwarded upstream](https://github.com/scrapy/scrapy/pull/5019)). * [#984809](https://bugs.debian.org/984809) filed against [`php8.0`](https://tracker.debian.org/pkg/php8.0). * [#985335](https://bugs.debian.org/985335) filed against [`cdebootstrap`](https://tracker.debian.org/pkg/cdebootstrap). * [#985448](https://bugs.debian.org/985448) filed against [`jalview`](https://tracker.debian.org/pkg/jalview). * Nilesh Patra: * [#985143](https://bugs.debian.org/985143) filed against [`bmtk`](https://tracker.debian.org/pkg/bmtk). * [#985144](https://bugs.debian.org/985144) filed against [`pigx-rnaseq`](https://tracker.debian.org/pkg/pigx-rnaseq). * [#985160](https://bugs.debian.org/985160) filed against [`simrisc`](https://tracker.debian.org/pkg/simrisc). * [#985210](https://bugs.debian.org/985210) filed against [`wordnet`](https://tracker.debian.org/pkg/wordnet). * [#985219](https://bugs.debian.org/985219) filed against [`xbs`](https://tracker.debian.org/pkg/xbs). * Vagrant Cascadian: * [#983832](https://bugs.debian.org/983832) filed against [`d-itg`](https://tracker.debian.org/pkg/d-itg). * [#983836](https://bugs.debian.org/983836) filed against [`crystal-facet-uml`](https://tracker.debian.org/pkg/crystal-facet-uml). * [#983902](https://bugs.debian.org/983902) filed against [`sendmail`](https://tracker.debian.org/pkg/sendmail). * [#984845](https://bugs.debian.org/984845) filed against [`sofia-sip`](https://tracker.debian.org/pkg/sofia-sip). * [#985187](https://bugs.debian.org/985187) filed against [`ffmpeg`](https://tracker.debian.org/pkg/ffmpeg). * [#985553](https://bugs.debian.org/985553) filed against [`dpkg`](https://tracker.debian.org/pkg/dpkg). ### Testing framework [![]({{ "/images/reports/2021-03/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/) The Reproducible Builds project operates a [Jenkins](https://jenkins.io/)-based testing framework that powers [`tests.reproducible-builds.org`](https://tests.reproducible-builds.org). This month, the following changes were made:
[![]({{ "/images/reports/2021-03/qubes.png#right" | relative_url }})](https://www.qubes-os.org/) * Frédéric Pierret ([Qubes-OS](https://www.qubes-os.org/)): * Improve the scripts to host `.buildinfo` files in a Debian-style "pool" directory structure. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/845b2b9d)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/abfbb8b5)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ff2462cb)] * Improve handling of temporary files. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5935769c)] * Create package sets in a public folder. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2b7b9017)] the * Merge a suite-specific script into the main one. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7e9f02e8)] * Fix an `awk` script. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a4c5bd77)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/49344dab)] * Holger Levsen: * Fix regular expression in host "health check" to correctly detect [Lintian](https://lintian.debian.org/) issues in [*diffoscope*](https://diffoscope.org/) builds [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fda0dd53)] as well as APT failures caused by broken dependencies [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c934c816)]. * Schedule `armhf` architecture *bullseye* packages in Debian more often than *unstable* as the release is near. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0d86554c)] * Further work on prototype Debian rebuilder tool to correct a typo in a *debrebuild* option [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/486bf74a)], to fail correctly even during when using "pipes" [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/779b14cf)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/84389356)] and make the debug output more readable in general [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5cab9517)]. * Handle temporary files files in the scripts to host `.buildinfo` files in a Debian-style "pool" directory structure [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f7290c04)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/40d7f445)] * Declare any `pool_buildinfos_suites` jobs as "zombies" jobs. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6b23fd57)] * Vagrant Cascadian: * Add a new `virt32a-armhf-rb.debian.net` and `virt64a-armhf-rb.debian.net` builders. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/dc874a83)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/679374a7)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ba3509ed)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3a975962)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b447f1f7)] * Re-enable `armhf` architecture nodes, now that they have built the *pbuilder* tarballs. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/106d19ed)] [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/08a8c8a0)] * Add a new package set for "[debian-on-mobile-maintainers](https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_maint_debian-on-mobile-maintainers.html)". [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8b093f76)] Elsewhere in our infrastructure, Mattia Rizzolo updated the [Mailman](https://list.org/) mailing list configuration to move the automated backups to run 10 minutes after midnight [[...](https://salsa.debian.org/reproducible-builds/rb-mailx-ansible/commit/5f300cc)] and to fix an [Ansible](https://www.ansible.com/) warning regarding Python `str` and` `int` types [[...](https://salsa.debian.org/reproducible-builds/rb-mailx-ansible/commit/87e4545)]. Lastly, build node maintenance was performed by Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b7ff8f8c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/eb3bec8a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2cd694ce)], Mattia Rizzolo [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2515ad8c)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fad3a9fc)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b5b62af6)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1f5b5550)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/46ec3c47)].
[![]({{ "/images/reports/2021-03/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * IRC: `#reproducible-builds` on `irc.oftc.net`. * Twitter ([@ReproBuilds](https://twitter.com/ReproBuilds)) & Mastodon ([@reproducible_builds@fosstodon.org](https://fosstodon.org/@reproducible_builds)) * Reddit: [/r/ReproducibleBuilds](https://reddit.com/r/reproduciblebuilds) * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)