---
layout: report
year: "2024"
month: "01"
title: "Reproducible Builds in January 2024"
draft: false
date: 2024-02-07 22:16:37
---

[![]({{ "/images/reports/2024-01/reproducible-builds.png#right" | relative_url }})]({{ "/" | relative_url }})

**Welcome to the January 2024 report from the [Reproducible Builds](https://reproducible-builds.org) project.** In these reports we outline the most important things that we have been up to over the past month. If you are interested in contributing to the project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.

---

### "How we executed a critical supply chain attack on PyTorch"

[John Stawinski](https://johnstawinski.com/) and [Adnan Khan](https://adnanthekhan.com/) published a lengthy blog post detailing [how they executed a supply-chain attack](https://johnstawinski.com/2024/01/11/playing-with-fire-how-we-executed-a-critical-supply-chain-attack-on-pytorch/) against [PyTorch](https://pytorch.org/), a popular machine learning platform "used by titans like Google, Meta, Boeing, and Lockheed Martin":

> Our exploit path resulted in the ability to upload malicious [PyTorch](https://pytorch.org/) releases to GitHub, upload releases to [Amazon Web Services], potentially add code to the main repository branch, backdoor PyTorch dependencies – the list goes on. **In short, it was bad. Quite bad.**

The attack pivoted on PyTorch's use of "[self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners)" as well as submitting a pull request to address a trivial typo in the project's `README` file to gain access to repository secrets and API keys that could subsequently be used for malicious purposes.

<br>

### New Arch Linux forensic filesystem tool

[![]({{ "/images/reports/2024-01/archlinux-userland-fs-cmp.png#right" | relative_url }})](https://github.com/kpcyrd/archlinux-userland-fs-cmp)

On our [mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/) this month, long-time Reproducible Builds developer *kpcyrd* [announced a new tool](https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003232.html) designed to forensically analyse [Arch Linux](https://archlinux.org/) filesystem images.

Called [`archlinux-userland-fs-cmp`](https://github.com/kpcyrd/archlinux-userland-fs-cmp), the tool is "supposed to be used from a rescue image (any Linux) with an Arch install mounted to, [for example], `/mnt`." Crucially, however, "at no point is any file from the mounted filesystem eval'd or otherwise executed. Parsers are written in a memory safe language."

More information about the tool can be found [on their announcement message](https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003232.html), as well as on the [tool's homepage](https://github.com/kpcyrd/archlinux-userland-fs-cmp). A [GIF of the tool in action](https://asciinema.org/a/MFefYEdvU2O5LlIzseQnyBky5) is also available.

<br>

### Issues with our `SOURCE_DATE_EPOCH` code?

Chris Lamb [started a thread on our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003225.html) summarising some potential problems with the source code snippet the Reproducible Builds project has been using to parse the [`SOURCE_DATE_EPOCH`]({{ "/docs/source-date-epoch/" | relative_url }}) environment variable:

> I'm not 100% sure who originally wrote this code, but it was probably sometime in the ~2015 era, and it must be in a huge number of codebases by now.
>
> Anyway, Alejandro Colomar was working on the shadow security tool and pinged me regarding some potential issues with the code. You can see this conversation [here](https://github.com/shadow-maint/shadow/commit/cb610d54b47ea2fc3da5a1b7c5a71274ada91371#r136407772).

Chris ends his message with a request that those with intimate or low-level knowledge of `time_t`, C types, overflows and the various parsing libraries in the C standard library (etc.) contribute with further info.

<br>

### Distribution updates

[![]({{ "/images/reports/2024-01/debian.png#right" | relative_url }})](https://debian.org/)

In Debian this month, Roland Clobus posted another [detailed update of the status of reproducible ISO images](https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003217.html) on our mailing list. In particular, Roland helpfully summarised that "all major desktops build reproducibly with *bullseye*, *bookworm*, *trixie* and *sid* provided they are built for a second time within the same DAK run (i.e. [within] 6 hours)". Additionally 7 of the 8 *bookworm* images from the [official download link](https://get.debian.org/cdimage/release/current-live/amd64/iso-hybrid/) build reproducibly at any later time.

In addition to this, three reviews of Debian packages were added, 17 were updated and 15 were removed this month adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html).

[![]({{ "/images/reports/2024-01/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)

Elsewhere, Bernhard posted another [monthly update](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/4AOWVPBX2OYQEUXTN3ORS6PJMUBAEWHS/) for his work elsewhere in openSUSE.

<br>

### Community updates

[![]({{ "/images/reports/2024-01/website.png#right" | relative_url }})]({{ "/" | relative_url }})

There were made a number of improvements to our website, including Bernhard M. Wiedemann fixing a number of typos of the term 'nondeterministic'.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/281bea1b)] and Jan Zerebecki adding a substantial and highly welcome section to our page about [`SOURCE_DATE_EPOCH`]({{ "/docs/source-date-epoch/" | relative_url }}) to document its interaction with distribution rebuilds.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3a3988bf)].

<br>

[![]({{ "/images/reports/2024-01/diffoscope.png#right" | relative_url }})](https://diffoscope.org/)

[diffoscope](https://diffoscope.org) is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions `254` and `255` to Debian but focusing on triaging and/or merging code from other contributors. This included adding support for comparing [eXtensible ARchive' (.XAR/.PKG)](https://en.wikipedia.org/wiki/Xar_(archiver)) files courtesy of Seth Michael Larson&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/241c92af)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/aca62e60)], as well considerable work from Vekhir in order to fix compatibility between various and subtle incompatible versions of the progressbar libraries in Python&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/1fae3be4)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/61394cc4)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/168f927c)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/828a33ab)]. Thanks!

<br>

### Reproducibility testing framework

[![]({{ "/images/reports/2024-01/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

The Reproducible Builds project operates a comprehensive testing framework (available at [tests.reproducible-builds.org](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen:

* [Debian](https://debian.org/)-related changes:

    * Reduce the number of `arm64` architecture workers from 24 to 16.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/79bf35a6d)]
    * Use [diffoscope](https://diffoscope.org/) from the Debian release being tested again.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/57fccefcf)]
    * Improve the handling when killing unwanted processes&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/95cf719fd)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4e278f7ce)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2328587ab)] and be more verbose about it, too&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1c4f6ffdf)].
    * Don't mark a job as 'failed' if process marked as 'to-be-killed' is already gone.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/63b812a1b)]
    * Display the architecture of builds that have been running for more than 48 hours.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/55a86760f)]
    * Reboot `arm64` nodes when they hit an OOM (out of memory) state.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d3f61eacd)]

* Package rescheduling changes:

    * Reduce IRC notifications to '1' when rescheduling due to package status changes.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5dcf67e88)]
    * Correctly set `SUDO_USER` when rescheduling packages.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/dd1f4b129)]
    * Automatically reschedule packages regressing to FTBFS (build failure) or FTBR (build success, but unreproducible).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8d145dc96)]

* [OpenWrt](https://openwrt.org/)-related changes:

    * Install the `python3-dev` and `python3-pyelftools` packages as they are now needed for the `sunxi` target.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/010155f3b)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7f4c47059)]
    * Also install the `libpam0g-dev` which is needed by some OpenWrt hardware targets.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c7efc35d4)]

* Misc:

    * As it's January, set the `real_year` variable to 2024&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2f59edd10)] and bump various copyright years as well&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ad04d1fab)].
    * Fix a large (!) number of spelling mistakes in various scripts.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e7bde6d9a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4cafbc58a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3fbd6ed7e)]
    * Prevent [Squid](https://www.squid-cache.org/) and [Systemd](https://systemd.io/) processes from being killed by the [kernel's OOM killer](https://www.kernel.org/doc/gorman/html/understand/understand016.html).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9efe90485)]
    * Install the `iptables` tool everywhere, else our custom `rc.local` script fails.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/481caed35)]
    * Cleanup the `/srv/workspace/pbuilder` directory on boot.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9b564c446)]
    * Automatically restart [Squid](https://www.squid-cache.org/) if it fails.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b408fbd38)]
    * Limit the execution of `chroot-installation` jobs to a maximum of 4 concurrent runs.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/71642c11d)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d3afa6d4c)]

Significant amounts of node maintenance was performed by Holger Levsen (eg.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f618266a0)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/11dc79d53)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d1cc288bd)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/715eda5ec)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7a909a1d2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fd362069e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0f82bda1f)] etc.) and Vagrant Cascadian (eg.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a06287c62)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3e4b2e507)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f5625f573)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5d8c7d32e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c366d93b5)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1726a5281)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/dee2b8bd2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3b86797d2)]).  Indeed, Vagrant Cascadian handled an extended power outage for the network running the Debian `armhf` architecture test infrastructure. This provided the incentive to replace the UPS batteries and consolidate infrastructure to reduce future UPS load.&nbsp;[[…](https://floss.social/@vagrantc/111853398019782907)]

Elsewhere in our infrastructure, however, Holger Levsen also adjusted the email configuration for `@reproducible-builds.org` to deal with a [new SMTP email attack](https://www.postfix.org/smtp-smuggling.html).&nbsp;[[...](https://salsa.debian.org/reproducible-builds/rb-mailx-ansible/commit/c1ab40a)]

<br>

### Upstream patches

The Reproducible Builds project tries to detects, dissects and fix as many (currently) unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

* Bernhard M. Wiedemann:

    * [`cython`](https://github.com/cython/cython/issues/5949) (nondeterminstic path issue)
    * [`deluge`](https://build.opensuse.org/request/show/1136411) (issue with modification time of `.egg` file)
    * [`gap-ferret`](https://build.opensuse.org/request/show/1136422), [`gap-semigroups`](https://build.opensuse.org/request/show/1136433) & [`gap-simpcomp`](https://build.opensuse.org/request/show/1136431) (nondeterministic `config.log` file)
    * [`grpc`](https://github.com/grpc/grpc/pull/35687) (filesystem ordering issue )
    * [`hub`](https://build.opensuse.org/request/show/1137377) (random)
    * [`kubernetes1.22`](https://build.opensuse.org/request/show/1137979) & [`kubernetes1.23`](https://build.opensuse.org/request/show/1137980) (sort-related issue)
    * [`kubernetes1.24`](https://build.opensuse.org/request/show/1136467) & [`kubernetes1.25`](https://build.opensuse.org/request/show/1136465) (`go -trimpath` vs random issue)
    * [`libjcat`](https://build.opensuse.org/request/show/1138082) (drop test files with random bytes)
    * [`luajit`](https://github.com/LuaJIT/LuaJIT/issues/1008) (Use new `d` option for deterministic bytecode output)
    * `meson` [[…](https://github.com/mesonbuild/meson/pull/12788)][[…](https://github.com/mesonbuild/meson/pull/12789)] (sort the results from Python filesystem call)
    * [`python-rjsmin`](https://build.opensuse.org/request/show/1137474) (drop [GCC instrumentation](https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html) artifacts)
    * [`qt6-virtualkeyboard+others`](https://bugreports.qt.io/browse/QTBUG-121643) (bug parallelism/race)
    * [`SoapySDR`](https://github.com/pothosware/SoapySDR/issues/428) (parallelism-related issue)
    * [`systemd`](https://github.com/systemd/systemd/pull/31080) (sorting problem)
    * [`warewulf`](https://build.opensuse.org/request/show/1137333) ([CPIO](https://www.gnu.org/software/cpio/) modification time issue, etc.)

* Chris Lamb:

    * [#1060254](https://bugs.debian.org/1060254) filed against [`mumble`](https://tracker.debian.org/pkg/mumble).

* James Addison:

    * [`guake`](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059917) ('Schroedinger' file due to race condition)
    * [`qhelpgenerator-qt5`](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059631) (timezone localization; fix also merged upstream for QT6)
    * [`sphinx`](https://github.com/sphinx-doc/sphinx/pull/11888) (search index `doctitle` sorting)

Separate to this, Vagrant Cascadian followed up with the relevant maintainers when reproducibility fixes were not included in newly-uploaded versions of the `mm-common` package in Debian — this was quickly fixed, however.&nbsp;[[…](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=977177#35)]

<br>

---

If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)

 * Mastodon: [@reproducible_builds](https://fosstodon.org/@reproducible_builds)

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)