---
layout: report
year: "2020"
month: "09"
title: "Reproducible Builds in September 2020"
draft: false
date: 2020-10-05 10:48:58
---

[![]({{ "/images/reports/2020-09/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)

**Welcome to the September 2020 report from the [Reproducible Builds](https://reproducible-builds.org) project.** In our monthly reports, we attempt to summarise the things that we have been up to over the past month, but if you are interested in contributing to the project, [please visit our main website]({{ "/" | relative_url }}).

[![]({{ "/images/reports/2020-09/ardc.png#right" | relative_url }})]({{ "https://ampr.org/" | relative_url }})

This month, the Reproducible Builds project was pleased to announce a donation from [Amateur Radio Digital Communications](https://ampr.org/) (ARDC) in support of its goals. ARDC's contribution will propel the Reproducible Builds project's efforts in ensuring the future health, security and sustainability of our increasingly digital society. [Amateur Radio Digital Communications](https://ampr.org/) (ARDC) is a non-profit which was formed to further research and experimentation with digital communications using radio, with a goal of advancing the state of the art of amateur radio and to educate radio operators in these techniques. You can [view the full announcement]({{ "/news/2020/09/25/ardc-sponsors-the-reproducible-builds-project/" | relative_url }}) as well as more information about ARDC [on their website](https://ampr.org/).

<br>

In August's report, we announced that [Jennifer Helsby](https://redshiftzero.github.io/) (*redshiftzero*) launched a new [reproduciblewheels.com](https://reproduciblewheels.com/) website to address the lack of reproducibility of [Python 'wheels'](https://pythonwheels.com/). This month, [Kushal Das](https://kushaldas.in/) posted a brief follow-up to provide an [update on reproducible *sources*](https://kushaldas.in/posts/reproducible-wheels-at-securedrop.html) as well.

[![]({{ "/images/reports/2020-09/threema.jpg#right" | relative_url }})](https://threema.ch/en/blog/posts/open-source-and-new-partner)

The [Threema](https://threema.ch/) privacy and security-oriented messaging application announced that "within the next months", their apps "will become fully open source, supporting reproducible builds":

> This is to say that anyone will be able to independently review Threema’s security and verify that the published source code corresponds to the downloaded app.

You can view [the full announcement](https://threema.ch/en/blog/posts/open-source-and-new-partner) on Threema's website.

## Events

Sadly, due to the unprecedented events in 2020, there will be [no in-person Reproducible Builds event this year](https://lists.reproducible-builds.org/pipermail/rb-general/2020-September/002045.html). However, the Reproducible Builds project intends to resume meeting regularly on IRC, starting on **[Monday, October 12th at 18:00 UTC](https://time.is/compare/1800_12_Oct_2020_in_UTC)** ([full announcement](https://lists.reproducible-builds.org/pipermail/rb-general/2020-September/002049.html)). The cadence of these meetings will probably be every two weeks, although this will be discussed and decided on at the first meeting. (An [editable agenda is available](https://pad.sfconservancy.org/p/reproducible-builds-meeting-agenda).)

[![]({{ "/images/reports/2020-09/isdd2020.png#right" | relative_url }})](https://www.eco.de/events/internet-security-days-2020/isdd-2020-agenda/#best_practises__aus_erfahrungen_lernen)

On 18th September, Bernhard M. Wiedemann gave a presentation in German titled [*Wie reproducible builds Software sicherer machen*](https://www.eco.de/events/internet-security-days-2020/isdd-2020-agenda/#best_practises__aus_erfahrungen_lernen) ("How reproducible builds make software more secure") at the [Internet Security Digital Days 2020](https://www.eco.de/events/internet-security-days-2020/) conference. ([View video](https://player.vimeo.com/video/451962745#t=429m24s).)

On Saturday 10th October, Morten Linderud will give a talk at [Arch Conf Online 2020](https://conf.archlinux.org/) on [*The State of Reproducible Builds*](https://pretalx.com/arch-conf-online-2020/talk/39BGNS/) in the Arch Linux distribution:

> The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.

During the [Reproducible Builds summit in Marrakesh]({{ "/events/Marrakesh2019/" | relative_url }}), [GNU Guix](https://guix.gnu.org), [NixOS](https://nixos.org) and [Debian](https://debian.org) were able to produce a bit-for-bit identical binary when building [GNU Mes](https://www.gnu.org/software/mes/), despite using three different major versions of GCC. Since the summit, additional work resulted in a bit-for-bit identical Mes binary using `tcc` and this month, [a fuller update was posted]({{ "/news/2019/12/21/reproducible-bootstrap-of-mes-c-compiler/" | relative_url }}) by the individuals involved.

<br>

## Development work

In [openSUSE](https://www.opensuse.org/), Bernhard M. Wiedemann published his [monthly Reproducible Builds status update](https://lists.opensuse.org/opensuse-factory/2020-10/msg00003.html).

#### [Debian](https://debian.org/)

[![]({{ "/images/reports/2020-09/debian.png#right" | relative_url }})](https://debian.org/)

[Chris Lamb](https://chris-lamb.co.uk) uploaded a number of Debian packages to address reproducibility issues that he had previously provided patches for, including [`cfingerd`](https://tracker.debian.org/pkg/cfingerd) ([#831021](https://bugs.debian.org/831021)), [`grap`](https://tracker.debian.org/pkg/grap) ([#870573](https://bugs.debian.org/870573)), [`splint`](https://tracker.debian.org/pkg/splint) ([#924003](https://bugs.debian.org/924003)) & [`schroot`](https://tracker.debian.org/pkg/schroot) ([#902804](https://bugs.debian.org/902804))

Last month, an issue was identified where a large number of Debian `.buildinfo` build certificates had been 'tainted' on the official Debian build servers, as [these environments had files underneath the `/usr/local/sbin` directory](https://bugs.debian.org/969084) to prevent the execution of system services during package builds. However, this month, Aurelien Jarno and Wouter Verhelst fixed this issue in varying ways, resulting in a special `policy-rcd-declarative-deny-all` package.

Building on Chris Lamb's [previous work on reproducible builds for Debian .ISO images](https://lists.reproducible-builds.org/pipermail/rb-general/2020-August/002018.html), Roland Clobus announced his work in progress on making the [Debian Live](https://wiki.debian.org/DebianLive/) images reproducible.&nbsp;[[...](https://lists.reproducible-builds.org/pipermail/rb-general/2020-September/002044.html)]

[Lucas Nussbaum](https://members.loria.fr/LNussbaum/) performed an archive-wide rebuild of packages to [test enabling the `reproducible=+fixfilepath` Debian build flag by default](https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20200921/012586.html). Enabling the `fixfilepath` feature will likely fix reproducibility issues in an estimated 500-700 packages. The test revealed only 33 packages (out of 30,000 in the archive) that [fail to build with `fixfilepath`](https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html). Many of those will be [fixed when the default LLVM/Clang](https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html) version [is upgraded](https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20200928/012594.html).

79 reviews of Debian packages were added, 23 were updated and 17 were removed this month adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb added and categorised a number of new issue types, including packages that [captures their build path via `quicktest.h`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/18a02c14) and [absolute build directories in documentation generated by Doxygen`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/2354fa49), etc.

[![]({{ "/images/reports/2020-09/intoto.png#right" | relative_url }})](https://in-toto.io/)

Lastly, Lukas Puehringer's uploaded a new version of the [*in-toto*](https://in-toto.io/) to Debian which was sponsored by Holger Levsen.&nbsp;[[...](https://tracker.debian.org/news/1173455/accepted-in-toto-050-1-source-into-unstable/)]


### [diffoscope](https://diffoscope.org)

[![]({{ "/images/reports/2020-09/diffoscope.svg#right" | relative_url }})](https://diffoscope.org)

[*diffoscope*](https://diffoscope.org) is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds too.

In September, [Chris Lamb](https://chris-lamb.co.uk) made the following changes to [diffoscope](https://diffoscope.org), including preparing and uploading versions `159` and `160` to Debian:

* New features:

    * Show "ordering differences" only in `strings(1)` output by applying the ordering check to all differences across the codebase.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/issues/216)]

* Bug fixes:

    * Mark some PGP tests that they require `pgpdump`, and check that the associated binary is actually installed before attempting to run it. ([#969753](https://bugs.debian.org/969753))
    * Don't raise exceptions when cleaning up after `guestfs` cleanup failure.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/aca4221)]
    * Ensure we check `FALLBACK_FILE_EXTENSION_SUFFIX`, otherwise we run `pgpdump` against all files that are recognised by `file(1)` as `data`.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/bab3e0e)]

* Codebase improvements:

    * Add some documentation for the `EXTERNAL_TOOLS` dictionary.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/c6a438a)]
    * Abstract out a variable we use a couple of times.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6bb06d2)]

* [diffoscope.org](https://diffoscope.org) website improvements:

    * Make the (long) demonstration GIF less prominent on the page.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/aae926e)]

In addition, Paul Spooren added support for automatically deploying Docker images.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/53d1aad)]

### [Website and documentation](https://reproducible-builds.org/)

[![]({{ "/images/reports/2020-09/website.png#right" | relative_url }})]({{ "/" | relative_url }})

This month, a number of updates to the [main Reproducible Builds website]({{ "/" | relative_url }}) and related documentation. [Chris Lamb](https://chris-lamb.co.uk) made the following changes:

* Update a few titles and the ordering of some top-level navigation elements.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9423f45)]
* Drafted, published and publicised [August's monthly report]({{ "/reports/2020-08/" | relative_url }}).
* Improve the documentation on how to signup to [Salsa](https://salsa.debian.org/).&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b4da392)]
* Add some more links to academic papers.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1b60222)]
* Also include the general news in our RSS feed [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/62370fc)] and drop including weekly reports from the RSS feed (they are never shown now that we have over 10 items) [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/18211d6)].
* Update ordering and location of various news and links to tarballs, etc.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/131a83d)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4aa0407)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5998e6f)]
* Kept [isdebianreproducibleyet.com](https://isdebianreproducibleyet.com) up to date.&nbsp;[[...](https://github.com/lamby/isdebianreproducibleyet.com/commits?author=lamby&since=2020-09-01T00:00:00Z&until=2020-10-01T00:00:00Z)]
* Worked with [Amateur Radio Digital Communications](https://ampr.org/) in order to announce their [generous sponsorship of the Reproducible Builds project]({{ "/news/2020/09/25/ardc-sponsors-the-reproducible-builds-project/" | relative_url }}).

In addition, Holger Levsen re-added the [documentation]({{ "/docs/" | relative_url }}) link to the top-level navigation [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/83e2a60)] and documented that the `jekyll-polyglot` package is required [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b8c3b09)]. Lastly, `diffoscope.org` and `reproducible-builds.org` were transferred to [Software Freedom Conservancy](https://sfconservancy.org/). Many thanks to Brett Smith from Conservancy, Jérémy Bobbio (*lunar*) and Holger Levsen for their help with transferring and to Mattia Rizzolo for initiating this.

#### Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:

* Bernhard M. Wiedemann:

    * [`cfn-python-lint`](https://github.com/aws-cloudformation/cfn-python-lint/issues/1705) (build failure)
    * [`clutter`](https://build.opensuse.org/request/show/835467) (avoid a random ID in HTML from `xsltproc`)
    * [`kubernetes`](https://github.com/kubernetes/kubernetes/issues/94628) (1-bit order in manual page)
    * [`libint`](https://github.com/evaleev/libint/pull/193) (merged, filesystem order)
    * [`libmysofa`](https://github.com/hoene/libmysofa/pull/139) (disable `-fprofile-arcs` and code coverage)
    * [`libnet`](https://github.com/libnet/libnet/pull/112) (merged, date)
    * [`libqb`](https://github.com/ClusterLabs/libqb/issues/414) (date / copyright)
    * [`libsemigroups`](https://build.opensuse.org/request/show/836070) (CPU detection)
    * [`nauty`](https://build.opensuse.org/request/show/838119) (CPU type detection)

* Chris Lamb:

    * [#970024](https://bugs.debian.org/970024) filed against [`cif2cell`](https://tracker.debian.org/pkg/cif2cell).
    * [#970278](https://bugs.debian.org/970278) filed against [`smartlist`](https://tracker.debian.org/pkg/smartlist).
    * [#970383](https://bugs.debian.org/970383) filed against [`evince`](https://tracker.debian.org/pkg/evince).
    * [#970498](https://bugs.debian.org/970498) filed against [`libiio`](https://tracker.debian.org/pkg/libiio).
    * [#970851](https://bugs.debian.org/970851) filed against [`check-pgbackrest`](https://tracker.debian.org/pkg/check-pgbackrest).
    * [#970908](https://bugs.debian.org/970908) filed against [`smartdns`](https://tracker.debian.org/pkg/smartdns).
    * [#971420](https://bugs.debian.org/971420) filed against [`jhbuild`](https://tracker.debian.org/pkg/jhbuild).

* *kpcyrd*:
    * [`git2-rs`](https://github.com/rust-lang/git2-rs/pull/619) (sort return ordering of `readdir(3)`)

* Vagrant Cascadian:

    * [#970874](https://bugs.debian.org/970874) filed against [`nix`](https://tracker.debian.org/pkg/nix).
    * [#970888](https://bugs.debian.org/970888) & [#970890](https://bugs.debian.org/970890) filed against [`fai`](https://tracker.debian.org/pkg/fai).
    * [#970893](https://bugs.debian.org/970893) filed against [`debdelta`](https://tracker.debian.org/pkg/debdelta).
    * [#971400](https://bugs.debian.org/971400) & [#971402](https://bugs.debian.org/971402) filed against [`vboot-utils`](https://tracker.debian.org/pkg/vboot-utils).
    * [#971408](https://bugs.debian.org/971408) filed against [`simde`](https://tracker.debian.org/pkg/simde).

Bernhard M. Wiedemann also reported issues in [`git2-rs`](https://github.com/rust-lang/git2-rs/issues/618), [`pyftpdlib`](https://github.com/giampaolo/pyftpdlib/issues/540), [`python-nbclient`](https://github.com/jupyter/nbclient/issues/114), [`python-pyzmq`](https://bugzilla.opensuse.org/show_bug.cgi?id=1176232) & [`python-sidpy`](https://github.com/pycroscopy/sidpy/issues/59).

## Testing framework

[![]({{ "/images/reports/2020-09/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

The Reproducible Builds project operates a [Jenkins](https://jenkins.io/)-based testing framework to power [`tests.reproducible-builds.org`](https://tests.reproducible-builds.org). This month, Holger Levsen made the following changes:

* [Debian](https://debian.org/):

    * Shorten the subject of "nodes have gone offline" notification emails.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2f6a1242)]
    * Also track bugs that have been [usertagged](https://wiki.debian.org/bugs.debian.org/usertags) with `usrmerge`.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/79ac64a9)]
    * Drop abort-related codepaths as that functionality has been removed from [Jenkins](https://jenkins.io/).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f5c1d675)]
    * Update the frequency we update base images and status pages.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/cf2497e7)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5461a909)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9c21f055)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6b2aca38)]

* [Status summary view page](https://tests.reproducible-builds.org/trbo.status.html):

    * Add support for monitoring `systemctl` status&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6fb89cbc)] and the number of [diffoscope](https://diffoscope.org/) processes&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a10c53c9)].
    * Show the total number of nodes&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/06aefcc6)] and colourise critical disk space situations&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/07297a9a)].
    * Improve the visuals with respect to vertical space.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4d7943c4)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a1e43681)]

* Debian rebuilder prototype:

    * Resume building random packages again&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e9a98ffa)] and update the frequency that packages are rebuilt.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/02487ea2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/84c959c3)]
    * Use `--no-respect-build-path` parameter until `sbuild` 0.81 is available.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/62e8b2ac)]
    * Treat the inability to locate some packages as a `debrebuild` problem, and not as a issue with the rebuilder itself.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/cb788349)]

* Arch Linux: [![]({{ "/images/reports/2020-09/archlinux.png#right" | relative_url }})](https://www.archlinux.org/)

    * Update various components to be compatible with Arch Linux's [move to the `xz` compression format](https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ccf1cb17)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ae55baaa)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/196f2f4b)]
    * Allow scheduling of old packages to catch up on the backlog.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/969d2b55)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c6013e6a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2ca91549)]
    * Improve formatting on the summary page.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4f8765db)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0f5f4aef)]
    * Update HTML pages once every hour, not every 30 minutes.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a55b0d2a)]
    * Use the [Ubuntu (!) GPG keyserver](https://keyserver.ubuntu.com/) to validate packages.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9a02d319)]

* System health checks:

    * Highlight important bad conditions in colour.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d099a852)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ef5ef2fe)]
    * Add support for detecting more problems, including Jenkins shutdown issues&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2ba109af)], failure to upgrade Arch Linux packages&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/38ba6311)], kernels with wrong permissions&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/585ecee1)], etc.

* Misc:

    * Delete old `schroot` sessions after 2 days, not 3.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/942da423)]
    * Use [sudo](https://www.sudo.ws/) to cleanup [*diffoscope*](https://diffoscope.org) `schroot` sessions.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d0a30cf1)]

In addition, *stefan0xC* fixed a query for unknown results in the handling of Arch Linux packages [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6fcde1d8)] and Mattia Rizzolo updated the template that notifies maintainers by email of their newly-unreproducible packages to ensure that it did not get caught in junk/spam folders [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b07571c)]. Finally, build node maintenance was performed by Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/910e526c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2ef5b44c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/eb28c92d)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e2e968fe)], Mattia Rizzolo [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ccbdc814)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0c6c256d)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7e1fdbfd)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0da623cf)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/542f1f80)].

<br>

If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)

 * Mastodon: [@reproducible_builds@fosstodon.org](https://fosstodon.org/@reproducible_builds)

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)