---
layout: report
year: "2021"
month: "04"
title: "Reproducible Builds in April 2021"
draft: false
date: 2021-05-09 15:12:39
---
**Welcome to the April 2021 report from the [Reproducible Builds](https://reproducible-builds.org) project!**
{: .lead}
[![]({{ "/images/reports/2021-04/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)
In these reports we try to the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. If you are interested in contributing to the project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.
[![]({{ "/images/reports/2021-04/ieee-paper.png#right" | relative_url }})](https://arxiv.org/abs/2104.06020)
A preprint of a paper by [Chris Lamb](https://chris-lamb.co.uk) and [Stefano Zacchiroli](https://upsilon.cc/~zack/) (which will shortly appear in [IEEE Software](https://www.computer.org/csdl/magazine/so)) has been made available on the *arXiv.org* service. Titled [*Reproducible Builds: Increasing the Integrity of Software Supply Chains*](https://arxiv.org/abs/2104.06020) ([PDF](https://arxiv.org/pdf/2104.06020)), the abstract of the paper contains the following:
> We first define the problem, and then provide insight into the challenges of making real-world software build in a "reproducible" manner-this is, when every build generates bit-for-bit identical results. Through the experience of the Reproducible Builds project making the Debian Linux distribution reproducible, we also describe the affinity between reproducibility and quality assurance (QA). [[...](https://arxiv.org/abs/2104.06020)]
Elsewhere on the internet, Igor Golovin on the [Kaspersky security blog](https://www.kaspersky.co.uk/blog/) reported that *APKPure*, an alternative app store for Android apps, began to distribute "an advertising SDK from an unverified source [that] turned out to be malicious" [[...](https://www.kaspersky.com/blog/infected-apkpure/39273/)]. Elaborating [elsewhere on the internet](https://securelist.com/apkpure-android-app-store-infected/101845/), Igor wrote that the malicious code had "much in common with the notorious [Triada malware](https://securelist.com/attack-on-zygote-a-new-twist-in-the-evolution-of-mobile-threats/74032/) and can perform a range of actions: from displaying and clicking ads to signing up for paid subscriptions and downloading other malware".
Closer to home, Jeremiah Orians wrote to [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) reporting that it is now possible to bootstrap the GCC compiler [without using the pre-generated Bison grammar files](https://lists.reproducible-builds.org/pipermail/rb-general/2021-April/002226.html), part of a broader attempt to provide a "reproducible, automatic [and] complete end-to-end bootstrap from a minimal number of binary seeds to a supported fully functioning operating system" [[...](https://github.com/fosslinux/live-bootstrap)]. In addition, Richard Clobus started a thread on [potential problems the `-Wl,--build-id=sha1` linker flag](https://lists.reproducible-builds.org/pipermail/rb-general/2021-April/002242.html) which can later be used when analysing core dumps and tracebacks. According to the [Red Hat Customer Portal](https://access.redhat.com/):
> Each executable or shared library built with Red Hat Enterprise Linux Server 6 or later is assigned a unique identification 160-bit [SHA-1](https://en.wikipedia.org/wiki/SHA-1) string, generated as a checksum of **selected parts of the binary**. This allows two builds of the same program on the same host to always produce consistent build-ids and binary content. (*emphasis added*)
[![]({{ "/images/reports/2021-04/fdroid.png#right" | relative_url }})](https://f-droid.org/)
Lastly, Felix C. Stegerman [reported on the latest release of `apksigcopier`](https://lists.reproducible-builds.org/pipermail/rb-general/2021-April/002237.html). [`apksigcopier`](https://github.com/obfusk/apksigcopier) is a tool to copy, extract and patch `.apk` signatures that is needed to facilitate reproducible builds on the [F-Droid](https://f-droid.org) Android application store and elsewhere. Holger Levsen subsequently sponsored an upload to Debian.
## Software development
### Distribution work
[![]({{ "/images/reports/2021-04/archlinux.png#right" | relative_url }})](https://archlinux.org/)
An issue was discovered in [Arch Linux](https://archlinux.org/) regarding packages that where previously considered reproducible. After some investigation, it was determined that the build's `CFLAGS` could vary between two previously 'reproducible' builds
The cause was attributed the fact that in Arch Linux, the [devtools](https://archlinux.org/packages/extra/any/devtools/) package determines the build configuration, but in the *development* branch it had been inadvertently copying the `makepkg.conf` file from the `pacman` package — the `devtools` version [had been fixed](https://github.com/archlinux/archlinux-repro/commit/9293d6f1e2197b63aaffb4936735eaebdd5e4620) in the recent release. This meant that when Arch Linux released or releases a `devtools` package with updated CFLAGS` (or similar), old packages could fail to build reproducibly as they would be reproduced in a different build environment.
To address this problem, Levente Polyak [sent a patch](https://lists.archlinux.org/pipermail/pacman-dev/2021-April/025044.html) to the `pacman` mailing list to include the version of `devtools` in the relevant `BUILDINFO` file. This means that the `repro` tool can now install the corresponding `makepkg.conf` file when attempting to validate a reproducible build.
[![]({{ "/images/reports/2021-04/debian.png#right" | relative_url }})](https://debian.org/)
In Debian, Frédéric Pierret continued working on [`debian.notset.fr`](https://debian.notset.fr/snapshot/), a partial copy of the [`snapshot.debian.org`](https://snapshot.debian.org/) "wayback machine" service for the Debian archive that is limited to the packages needed to rebuild the *bullseye* distribution on the `amd64` architecture. This is to workaround some perceived limitations of `snapshot.debian.org`. Since [last month]({{ "/reports/2021-03/" | relative_url }}), the service covers from mid-2020 onwards, and [request was made](https://rt.debian.org/Ticket/Display.html?id=8547) to the [Debian sysadmin team](https://wiki.debian.org/Teams/DSA) to obtain better access to `snapshot.debian.org` in order to further accelerate the initial seeding. In addition, the service supports now more endpoints in the API ([full documentation](https://github.com/fepitre/debian-snapshot-mirror#api)), including a [`timestamp`](https://debian.notset.fr/snapshot/by-timestamp/) endpoint to track the sync in a machine-readable way.
Twenty-one reviews of Debian packages were performed, nine were updated and sixteen were removed this month adding to [our large taxonomy of identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). A number of issue types have been updated too, including removing the `random_order_in_javahelper_substvars` issue type [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/7a62f832)], but also the addition of a new `timestamps_in_pdf_generated_by_libreoffice` toolchain issue by Chris Lamb [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/b1a2295c)].
[![]({{ "/images/reports/2021-04/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)
Lastly, Bernhard M. Wiedemann posted his [monthly reproducible builds status report](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/ZGSD5CLQCSJXKK5LRWICFPZ4HZOREIJX/) for the [openSUSE](https://www.opensuse.org/) distribution.
#### Upstream patches
* Bernhard M. Wiedemann:
* [`digikam`](https://invent.kde.org/graphics/digikam/-/merge_requests/58) (date issue)
* [`k9s`](https://github.com/derailed/k9s/pull/1099) (date issue)
* [`librsb`](https://bugzilla.opensuse.org/show_bug.cgi?id=1184750) (memory layout issue)
* [`nDPI`](https://github.com/ntop/nDPI/pull/1176) (date issue)
* Verified openSUSE Leap 15.3 and SLES-15-SP3 binaries, and submitted [several reproducibility fixes](http://rb.zq1.de/sle/15.3/report.txt).
* Chris Lamb:
* [#971527](https://bugs.debian.org/971527) filed against [`libsass-python`](https://tracker.debian.org/pkg/rust-configparser) ([merged upstream](https://github.com/sass/libsass-python/pull/319#issuecomment-811671636)).
* [#986877](https://bugs.debian.org/986877) filed against [`rust-configparser`](https://tracker.debian.org/pkg/rust-configparser).
* Morten Linderud:
* [`ronn-ng`](https://github.com/apjanke/ronn-ng/pull/76) (date-related issue)
* Nilesh Patra:
* [#986601](https://bugs.debian.org/986601) filed against [`libjama`](https://tracker.debian.org/pkg/libjama).
* [#986642](https://bugs.debian.org/986642) filed against [`weka`](https://tracker.debian.org/pkg/weka).
* [#986738](https://bugs.debian.org/986738) filed against [`starlink-ast`](https://tracker.debian.org/pkg/starlink-ast).
### [*diffoscope*](https://diffoscope.org)
[![]({{ "/images/reports/2021-04/diffoscope.svg#right" | relative_url }})](https://diffoscope.org)
[*diffoscope*](https://diffoscope.org) is the Reproducible Builds project in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it provides human-readable diffs from many kinds of binary formats. This month, [Chris Lamb](https://chris-lamb.co.uk) made a number of changes including releasing [version 172](https://diffoscope.org/news/diffoscope-172-released/) and [version 173](https://diffoscope.org/news/diffoscope-173-released/):
* Add support for showing annotations in PDF files. ([#249](https://salsa.debian.org/reproducible-builds/issues/-/249))
* Move to the `assert_diff` helper in `test_pdf.py. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6be044d)]
In addition, Mattia Rizzolo attempted to make the testsuite pass with [`file(1)`](http://darwinsys.com/file/) version 5.40 [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/7bf04a6)] and Zachary T. Welch updated the `__init__` magic method of the `Difference` class to demote the `unified_diff` argument to a Python 'kwarg' [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/a3bfba0)].
### Website and documentation
[![]({{ "/images/reports/2021-04/website.png#right" | relative_url }})](https://reproducible-builds.org/)
Quite a few changes were made to the [main Reproducible Builds website and documentation](https://reproducible-builds.org/) this month, including:
* Chris Lamb:
* Highlight our mailing list on the [*Contribute*]({{ "/contribute/" | relative_url }}). page [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6532b31)]
* Add a noun (and drop an unnecessary full-stop) on the [landing page]({{ "/" | relative_url }}). [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2d8e6b9)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d0f245e)]
* Correct a reference to the `date` metadata attribute on reports, restoring the display of months on the homepage. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/030724d)]
* Correct a typo of "instalment" within a previous news entry. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/34aef72)]
* Added a conspicuous "draft" banner to unpublished blog posts in order to match the report draft banner. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/518daa1)]
* Mattia Rizzolo:
* Various improvements to the sponsors pages. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/599891c)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/c3139c3)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/c08ae3a)]
* Add the project's platinum-level sponsors to the homepage. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0573ffc)]
* Use a CSS class instead of specifying an inline `style` HTML attribute. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/517f759)]
### Testing framework
[![]({{ "/images/reports/2021-04/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)
The Reproducible Builds project operates a [Jenkins](https://jenkins.io/)-based testing framework that powers [`tests.reproducible-builds.org`](https://tests.reproducible-builds.org). This month, the following changes were made:
* Holger Levsen:
* Debian:
* Update `README` to reflect that Debian *buster* is now the 'stable' distribution. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/12a2449e)]
* Support fully-qualified domain names in the powercycle script for the `armhf` architecture. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/75e4db31)]
* Improve the handling of node names (etc.) for `armhf` nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/912e00e0)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f3240cd0)]
* Improve the detection and classification of packages maintained by the [Debian accessibility team](https://wiki.debian.org/accessibility-devel). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/419a81b0)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/139862c1)]
* Count the number configured *armhf* nodes correctly by ignoring comments at the end of line. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/275744db)]
* Health checks
* Improve checks for broken [OpenSSH](https://www.openssh.com/) ports. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ce076847)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1148928b)]
* Detect failures of NetBSD's `make release`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/08ab9c57)]
* Catch another log message variant that specifies a host is running an outdated kernel. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a4b56e9d)]
* Automatically restart failed `systemd-journal-flush` [systemd](https://www.freedesktop.org/wiki/Software/systemd/) services. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1373ed9c)]
* Other:
* Update FreeBSD to 13.0. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5a02807c)]
* Be less picky about "too many" installed kernels on hosts which have large enough `/boot` partition. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/39f85590)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5026edc7)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/78bb142b)]
* Unify some IRC output. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3efd89f7)]
* Mattia Rizzolo:
* Fix a regular expression in automatic log-parsing routines. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7b6dec23)]
* Vagrant Cascadian:
* Add some new `armhf` architecture build nodes, `virt32b` and `virt64b`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/18872c83)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5365d6a9)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d1a775ba)]
* Rearrange `armhf` build jobs to only use active nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4a5083b5)]
* Add a health check for broken [OpenSSH](https://www.openssh.com/) ports on `virt32a`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8b0bf2d2)]
* Mark which `armhf` architecture jobs are not systematically varying 32-bit and 64-bit kernels. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a2e0c796)]
* Disable creation of Debian *stretch* build tarballs and update the `README` file to mention *bullseye* instead. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6b26c871)]
Finally, build node maintenance was performed by Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6f5e8470)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fa4dcfb1)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a42ad2cc)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3d1ecade)], Mattia Rizzolo [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/25e20ec4)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/568bd62d)] [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b6865d93)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/534bc73c)].
[![]({{ "/images/reports/2021-04/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)
If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:
* Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)
* IRC: `#reproducible-builds` on `irc.oftc.net`.
* Twitter ([@ReproBuilds](https://twitter.com/ReproBuilds)) & Mastodon ([@reproducible_builds@fosstodon.org](https://fosstodon.org/@reproducible_builds))