---
layout: report
year: "2023"
month: "10"
title: "Reproducible Builds in October 2023"
draft: false
date: 2023-11-11 12:39:23
---

[![]({{ "/images/reports/2023-10/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)

**Welcome to the October 2023 report from the [Reproducible Builds](https://reproducible-builds.org) project.** In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.

---

### Reproducible Builds Summit 2023

[![]({{ "/images/reports/2023-10/summit.jpg#right" | relative_url }})]({{ "/events/hamburg2023/" | relative_url }})

Between October 31st and November 2nd, we held our [seventh Reproducible Builds Summit]({{ "/events/hamburg2023/" | relative_url }}) in Hamburg, Germany!

Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort, and this instance was no different.

During this enriching event, participants had the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. A number of concrete outcomes from the summit will documented in the report for November 2023 and elsewhere.

Amazingly the agenda and all notes from all sessions are [already online](https://reproducible-builds.org/events/hamburg2023/agenda/).

The Reproducible Builds team would like to thank our event sponsors who include [Mullvad VPN](https://mullvad.net/), [openSUSE](https://www.opensuse.org/), [Debian](https://www.debian.org/), [Software Freedom Conservancy](https://sfconservancy.org/), [Allotropia](https://www.debian.org/) and [Aspiration Tech](https://aspirationtech.org/).

<br>

### Reflections on *Reflections on Trusting Trust*

[![]({{ "/images/reports/2023-10/nih.png#right" | relative_url }})](https://research.swtch.com/nih)

[Russ Cox](https://swtch.com/~rsc/) posted a [fascinating article on his blog](https://research.swtch.com/nih) prompted by the fortieth anniversary of Ken Thompson's award-winning paper, [*Reflections on Trusting Trust*](https://dl.acm.org/doi/pdf/10.1145/358198.358210):

> […] In March 2023, Ken gave the closing keynote [and] during the Q&A session, someone jokingly asked about the Turing award lecture, specifically “can you tell us right now whether you have a backdoor into every copy of *gcc* and Linux still today?”

Although Ken reveals (or at least *claims*!) that he has no such backdoor, he does admit that he has the actual code… which Russ requests and subsequently dissects in great but accessible detail.

<br>

### Ecosystem factors of reproducible builds

[![]({{ "/images/reports/2023-10/time-to-fix-paper.png#right" | relative_url }})](https://mcis.cs.queensu.ca/publications)

Rahul Bajaj, Eduardo Fernandes, Bram Adams and Ahmed E. Hassan from the [Maintenance, Construction and Intelligence of Software (MCIS)](https://mcis.cs.queensu.ca) laboratory within the [School of Computing](https://cs.queensu.ca/), [Queen's University](https://www.queensu.ca/) in Ontario, Canada have published a paper on the "*Time to fix, causes and correlation with external ecosystem factors*" of unreproducible builds.

The authors compare various response times within the [Debian](https://debian.org/) and [Arch Linux](https://archlinux.org/) distributions including, for example:

> Arch Linux packages become reproducible a median of 30 days quicker when compared to Debian packages, while Debian packages remain reproducible for a median of 68 days longer once fixed.

A [full PDF of their paper](https://mcis.cs.queensu.ca/publications/2023/emse_rahul.pdf) is available online, as are many other interesting papers on [MCIS'](https://mcis.cs.queensu.ca/publications) publication page.

<br>

### NixOS installation image reproducible

[![]({{ "/images/reports/2023-10/nixos.png#right" | relative_url }})](https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756)

On the [NixOS Discourse instance](https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756), Arnout Engelen (*raboof*) announced that NixOS have created an independent, bit-for-bit identical rebuilding of the `nixos-minimal` image that is used to install NixOS. [In their post](https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756), Arnout details what exactly can be reproduced, and even includes some of the history of this endeavour:

> You may remember a [2021 announcement](https://discourse.nixos.org/t/nixos-unstable-s-iso-minimal-x86-64-linux-is-100-reproducible/13723) that the minimal ISO was 100% reproducible. While back then we successfully tested that all packages that were needed to build the ISO were individually reproducible, actually rebuilding the ISO still introduced differences. This was due to [some remaining problems](https://github.com/NixOS/nixpkgs/issues/125380) in the hydra cache and the way the ISO was created. By the time we fixed those, regressions had popped up (notably an upstream problem in Python 3.10), and it isn’t until this week that we were back to having everything reproducible and being able to validate the complete chain.

Congratulations to NixOS team for reaching this important milestone! Discussion about this announcement [can be found underneath the post](https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756#post_2) itself, as well as [on Hacker News](https://news.ycombinator.com/item?id=38057591).

<br>

### CPython source tarballs now reproducible

[![]({{ "/images/reports/2023-10/python-logo.png#right" | relative_url }})](https://github.com/python/release-tools/pull/62)

[Seth Larson](https://sethmlarson.dev/) published a blog post [investigating the reproducibility of the CPython source tarballs](https://sethmlarson.dev/security-developer-in-residence-weekly-report-14). Using [*diffoscope*](https://diffoscope.org/), *reprotest* and other tools, Seth documents his work that led to [a pull request to make these files reproducible](https://github.com/python/release-tools/pull/62) which was merged by [Łukasz Langa](https://lukasz.langa.pl/).

<br>

### New `arm64` hardware from Codethink

[![]({{ "/images/reports/2023-10/codethink.png#right" | relative_url }})](https://www.codethink.co.uk/)

Long-time sponsor of the project, [Codethink](https://www.codethink.co.uk/), have generously replaced our old "Moonshot-Slides", which they have generously hosted since 2016 with new [KVM](https://linux-kvm.org/page/Main_Page)-based `arm64` hardware. Holger Levsen integrated these new nodes to the [Reproducible Builds' continuous integration](https://tests.reproducible-builds.org/) framework.

<br>

### Community updates

On our [mailing list during October 2023](https://lists.reproducible-builds.org/pipermail/rb-general/2023-October/thread.html) there were a number of threads, including:

* Vagrant Cascadian continued a thread about the implementation details of a "snapshot" archive server required for reproducing previous builds.&nbsp;[[...](https://lists.reproducible-builds.org/pipermail/rb-general/2023-October/003086.html)]

* Akihiro Suda shared an update on [BuildKit](https://github.com/moby/buildkit), a toolkit for building [Docker](https://www.docker.com/) container images. Akihiro links to a interesting talk they recently gave at [DockerCon](https://dockercon.com/) titled [*Reproducible builds with BuildKit for software supply-chain security*](https://medium.com/nttlabs/dockercon-2023-reproducible-builds-with-buildkit-for-software-supply-chain-security-0e5aedd1aaa7).

* Alex Zakharov started a thread discussing and proposing fixes for various tools that create [`ext4`](https://en.wikipedia.org/wiki/Ext4) filesystem images.&nbsp;[[...](https://lists.reproducible-builds.org/pipermail/rb-general/2023-October/003098.html)]

Elsewhere, Pol Dellaiera made a number of improvements to our website, including fixing typos and links&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7f3e9550)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8ab7459c)], adding a [NixOS "Flake" file](https://nixos.wiki/wiki/Flakes)&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0c1a61eb)] and sorting our [publications page]({{ "/docs/publications/" | relative_url }}) by date&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d052569b)].

Vagrant Cascadian presented [*Reproducible Builds All The Way Down*](https://www.osfc.io/2023/talks/reproducible-builds-all-the-way-down/) at the [Open Source Firmware Conference](https://www.osfc.io/).

<br>

### Distribution work

[![]({{ "/images/reports/2023-10/debian.png#right" | relative_url }})](https://debian.org/)

*distro-info* is a Debian-oriented tool that can provide information about Debian (and Ubuntu) distributions such as their codenames (eg. *bookworm*) and so on. This month, Benjamin Drung uploaded a new version of *distro-info* that added support for the [`SOURCE_DATE_EPOCH` environment variable]({{ "/specs/source-date-epoch/" | relative_url }}) in order to close bug [#1034422](https://tracker.debian.org/distro-info). In addition, 8 reviews of packages were added, 74 were updated and 56 were removed this month, all adding to our [knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html).

Bernhard M. Wiedemann published another [monthly report about reproducibility within openSUSE](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/4QTSQCYBMF6QZYWIB63T46ILLTVGVMMJ/).

<br>

### Software development

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

 * Bernhard M. Wiedemann:

    * [`edje_cc`](https://git.enlightenment.org/enlightenment/efl/issues/41) (race condition)
    * [`elasticsearch`](https://github.com/elastic/elasticsearch-py/issues/2320) (build failure)
    * [`erlang-retest`](https://build.opensuse.org/request/show/1116208) (embedded `.zip` timestamp)
    * [`fdo-client`](https://bugzilla.opensuse.org/show_bug.cgi?id=1216293) (embeds private keys)
    * [`fftw3`](https://github.com/FFTW/fftw3/issues/337) (random ordering)
    * [`gsoap`](https://sourceforge.net/p/gsoap2/patches/185/) (date issue)
    * [`gutenprint`](https://sourceforge.net/p/gimp-print/source/merge-requests/9/) (date)
    * [`hub/golang`](https://github.com/golang/go/issues/63851) (embeds random build path)
    * [`Hyprland`](https://github.com/hyprwm/Hyprland/pull/3550) (filesystem issue)
    * [`kitty`](https://github.com/kovidgoyal/kitty/pull/6685) (sort-related issue, `.tar` file embeds modification time)
    * [`libpinyin`](https://github.com/libpinyin/libpinyin/issues/162) (ASLR)
    * [`maildir-utils`](https://github.com/djcb/mu/pull/2569) (date embedded in copyright)
    * [`mame`](https://github.com/mamedev/mame/pull/11651) (order-related issue)
    * [`mingw32-binutils`](https://build.opensuse.org/request/show/1116036) & [`mingw64-binutils`](https://build.opensuse.org/request/show/1116040) (date)
    * [`MooseX`](https://github.com/maros/MooseX-App/pull/71) (date from perl-MooseX-App)
    * [`occt`](https://build.opensuse.org/request/show/1119524) (sorting issue)
    * [`openblas`](https://build.opensuse.org/request/show/1118201) (embeds CPU count)
    * [`OpenRGB`](https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3675) ([corruption-related issue](https://gitlab.com/CalcProgrammer1/OpenRGB/-/merge_requests/2103))
    * [`python-numpy`](https://bugzilla.opensuse.org/show_bug.cgi?id=1216458) (random file names)
    * [`python-pandas`](https://build.opensuse.org/request/show/1117743) (FTBFS)
    * [`python-quantities`](https://build.opensuse.org/request/show/1117898) (date)
    * [`python3-pyside2`](https://bugreports.qt.io/browse/PYSIDE-2508) (order)
    * [`qemu`](https://build.opensuse.org/request/show/1121011) (date and Sphinx issue)
    * [`qpid`](https://github.com/apache/qpid-proton/pull/411) (sorting problem)
    * [`rakudo`](https://github.com/rakudo/rakudo/pull/5426) (filesystem ordering issue)
    * [`SLOF`](https://gitlab.com/qemu-project/SLOF/-/merge_requests/1) (date-related issue)
    * [`spack`](https://build.opensuse.org/request/show/1118130) (CPU counting issue)
    * [`xemacs-packages`](https://build.opensuse.org/request/show/1119260) (date-related issue)

* Chris Lamb:

    * [#1053353](https://bugs.debian.org/1053353) filed against [`dacite`](https://tracker.debian.org/pkg/dacite).
    * [#1053356](https://bugs.debian.org/1053356) filed against [`rtpengine`](https://tracker.debian.org/pkg/rtpengine).

In addition, Chris Lamb fixed an issue in [*diffoscope*](https://diffoscope.org), where if the equivalent of `file -i` returns `text/plain`, fallback to comparing as a text file. This was originally filed as Debian bug [#1053668](https://bugs.debian.org/1053668)) by Niels Thykier.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/81c68d7b)] This was then uploaded to Debian (and elsewhere) as version `251`.

<br>

### Reproducibility testing framework

[![]({{ "/images/reports/2023-10/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

The Reproducible Builds project operates a comprehensive testing framework (available at [tests.reproducible-builds.org](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen:

* Debian-related changes:

    * Refine the handling of package blacklisting, such as sending blacklisting notifications to the `#debian-reproducible-changes` IRC channel.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/07bd72f45)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/987448aba)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8fdd35cff)]
    * Install `systemd-oomd` on all Debian *bookworm* nodes (re. Debian bug [#1052257](https://bugs.debian.org/1052257)).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/08785fdf3)]
    * Detect more cases of failures to delete `schroots`.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b32f37dbf)]
    * Document various bugs in *bookworm* which are (currently) being manually worked around.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/67612f231)]

* Node-related changes:

    * Integrate the new `arm64` machines from [Codethink](https://www.codethink.co.uk/).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4411b061e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4e710ec37)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/916706239)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f7b7aa5c0)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/df2deec90)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/05b2bcd34)]
    * Improve various node cleanup routines.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d41fdb63f)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/08920dc6a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7c174dc2a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/357c8120d)]
    * General node maintenance.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/93765e006)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b70f0f06b)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c3d527015)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1812e9c6d)]

* Monitoring-related changes:

    * Remove unused [Munin](https://munin-monitoring.org/) monitoring plugins.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/db72c0b34)]
    * Complain less visibly about "too many" installed kernels.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b75d6f5e4)]

* Misc:

    * Enhance the firewall handling on Jenkins nodes.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/08bf35af2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5695c235a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b02d22f00)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/aaa68e7bf)]
    * Install the `fish` shell everywhere.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f460d6b44)]


In addition, Vagrant Cascadian added some packages and configuration for snapshot experiments.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/896f12ede)]

<br>

---

If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)

 * Mastodon: [@reproducible_builds](https://fosstodon.org/@reproducible_builds)

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)