---
layout: report
year: "2024"
month: "03"
title: "Reproducible Builds in March 2024"
draft: false
date: 2024-04-11 16:49:07
---

[![]({{ "/images/reports/2024-03/reproducible-builds.png#right" | relative_url }})]({{ "/" | relative_url }})

**Welcome to the March 2024 report from the [Reproducible Builds](https://reproducible-builds.org) project!** In our reports, we attempt to outline what we have been up to over the past month, as well as mentioning some of the important things happening more generally in software supply-chain security. As ever, if you are interested in contributing to the project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.

**Table of contents:**

0. [Arch Linux minimal container userland now 100% reproducible](#arch-linux-minimal-container-userland-now-100-reproducible)
0. [Validating Debian’s build infrastructure after the XZ backdoor](#validating-debians-build-infrastructure-after-the-xz-backdoor)
0. [Making Fedora Linux (more) reproducible](#making-fedora-linux-more-reproducible)
0. [Increasing Trust in the Open Source Supply Chain with Reproducible Builds and Functional Package Management](#increasing-trust-in-the-open-source-supply-chain-with-reproducible-builds-and-functional-package-management)
0. [Software and source code identification with GNU Guix and reproducible builds](#software-and-source-code-identification-with-gnu-guix-and-reproducible-builds)
0. [Two new Rust-based tools for post-processing determinism](#two-new-rust-based-tools-for-post-processing-determinism)
0. [Distribution work](#distribution-work)
0. [Mailing list highlights](#mailing-list-highlights)
0. [Website updates](#website-updates)
0. [Delta chat clients now reproducible](#delta-chat-clients-now-reproducible)
0. [diffoscope updates](#diffoscope)
0. [Upstream patches](#upstream-patches)
0. [Reproducibility testing framework](#reproducibility-testing-framework)

---

### Arch Linux minimal container userland now 100% reproducible

[![]({{ "/images/reports/2024-03/archlinux.png#right" | relative_url }})](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003301.html)

In remarkable news, Reproducible builds developer *kpcyrd* reported that that the [Arch Linux](https://archlinux.org/) "minimal container userland" [is now 100% reproducible](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003301.html) after work by developers *dvzv* and *Foxboron* on the one remaining package. This represents a "real world", widely-used Linux distribution being reproducible.

Their post, which *kpcyrd* suffixed with the question "now what?", continues on to outline some potential next steps, including validating whether the container image itself could be reproduced bit-for-bit. The post, which was itself a followup for an [Arch Linux update earlier in the month](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003291.html), generated a [significant number of replies](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/thread.html#3301).

<br>

### Validating Debian's build infrastructure after the XZ backdoor

[![]({{ "/images/reports/2024-03/debian.png#right" | relative_url }})](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003321.html)

From our [mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month, Vagrant Cascadian [wrote about](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003321.html) being asked about trying to perform concrete reproducibility checks for recent Debian security updates, in an attempt to gain some confidence about Debian's build infrastructure given that they performed builds in environments running the [high-profile XZ vulnerability](https://lwn.net/Articles/967866/).

Vagrant reports (with some caveats):

> So far, I have not found any reproducibility issues; everything I tested I was able to get to build bit-for-bit identical with what is in the
Debian archive.

That is to say, reproducibility testing permitted Vagrant and Debian to claim with some confidence that builds performed when this vulnerable version of XZ was installed were not interfered with.

<br>

### Making Fedora Linux (more) reproducible

[![]({{ "/images/reports/2024-03/5c4gfXVPAbU.jpg#right" | relative_url }})](https://www.youtube.com/watch?v=5c4gfXVPAbU)

In March, Davide Cavalca gave a talk at the [2024 Southern California Linux Expo](https://www.socallinuxexpo.org/scale/21x) (aka *SCALE 21x*) about the ongoing effort to [make the Fedora Linux distribution reproducible](https://www.socallinuxexpo.org/scale/21x/presentations/making-fedora-linux-more-reproducible).

Documented in [more detail on Fedora's website](https://docs.fedoraproject.org/en-US/reproducible-builds/), the talk touched on topics such as the specifics of implementing reproducible builds in Fedora, the challenges encountered, the current status and what's coming next. ([YouTube video](https://www.youtube.com/watch?v=5c4gfXVPAbU))

<br>

### "*Increasing Trust in the Open Source Supply Chain with Reproducible Builds and Functional Package Management*"

[![]({{ "/images/reports/2024-03/hal-04482192.png#right" | relative_url }})](https://hal.science/hal-04482192)

Julien Malka published a brief but interesting paper in the [HAL open archive](https://en.wikipedia.org/wiki/HAL_(open_archive)) on [*Increasing Trust in the Open Source Supply Chain with Reproducible Builds and Functional Package Management*](https://hal.science/hal-04482192):

> Functional package managers (FPMs) and reproducible builds (R-B) are technologies and methodologies that are conceptually very different from the traditional software deployment model, and that have promising properties for software supply chain security. This thesis aims to evaluate the impact of FPMs and R-B on the security of the software supply chain and propose improvements to the FPM model to further improve trust in the open source supply chain. [PDF](https://hal.science/hal-04482192/document)

Julien's paper poses a number of research questions on how the model of distributions such as [GNU Guix](https://guix.gnu.org/) and [NixOS](https://nixos.org/) can "be leveraged to further improve the safety of the software supply chain", etc.

<br>

### Software and source code identification with [GNU Guix](https://guix.gnu.org/) and reproducible builds

In a long line of commendably detailed blog posts, Ludovic Courtès, Maxim Cournoyer, Jan Nieuwenhuizen and Simon Tournier have together published two interesting posts on the [GNU Guix blog](https://guix.gnu.org/en/blog/) this month. In early March, Ludovic Courtès, Maxim Cournoyer, Jan Nieuwenhuizen and Simon Tournier wrote about [software and source code identification](https://guix.gnu.org/en/blog/2024/identifying-software/) and how that might be performed using Guix, rhetorically posing the questions: "What does it take to 'identify software'? How can we tell what software is running on a machine to determine, for example, what security vulnerabilities might affect it?"

[![]({{ "/images/reports/2024-03/safety-last.jpg#right" | relative_url }})](https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/)

Later in the month, Ludovic Courtès wrote a solo post describing adventures on [the quest for long-term reproducible deployment](https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/). Ludovic's post touches on GNU Guix's aim to support "time travel", the ability to reliably (and reproducibly) revert to an earlier point in time, employing the iconic image of Harold Lloyd hanging off the clock in [*Safety Last!*](https://en.wikipedia.org/wiki/Safety_Last!) (1925) to poetically illustrate both the slapstick nature of current modern technology and the gymnastics required to navigate hazards of our own making.

<br>

### Two new Rust-based tools for post-processing determinism

Zbigniew Jędrzejewski-Szmek announced [*add-determinism*](https://github.com/keszybz/add-determinism), a work-in-progress reimplementation of the Reproducible Builds project's own [*strip-nondeterminism*](https://salsa.debian.org/reproducible-builds/strip-nondeterminism) tool in the [Rust programming language](https://www.rust-lang.org/), intended to be used as a post-processor in RPM-based distributions such as [Fedora](https://fedoraproject.org/)

In addition, [Yossi Kreinin](https://yosefk.com/) published a [blog post titled "*refix: fast, debuggable, reproducible builds*"](https://yosefk.com/blog/refix-fast-debuggable-reproducible-builds.html) that describes a tool that post-processes binaries in such a way that they are still debuggable with [gdb](https://sourceware.org/gdb/), etc.. Yossi post details the motivation and techniques behind the (fast) performance of the tool.

<br>

### Distribution work

[![]({{ "/images/reports/2024-03/debian.png#right" | relative_url }})](https://debian.org/)

In Debian this month, since the testing framework no longer varies the [build path]({{ "/docs/build-path/" | relative_url }}), James Addison performed a [bulk downgrade of the bug severity](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003257.html) for issues filed with a level of `normal` to a new level of `wishlist`. In addition, 28 reviews of Debian packages were added, 38 were updated and 23 were removed this month adding to ever-growing [knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). As part of this effort, a number of issue types were updated, including Chris Lamb adding a new `ocaml_include_directories` toolchain issue [[…](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/a052c30f)] and James Addison adding a new `filesystem_order_in_java_jar_manifest_mf_include_resource` issue [[…](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/cc94c935)] and updating the  `random_uuid_in_notebooks_generated_by_nbsphinx` to reference a relevant discussion thread [[…](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/55497f89)].

In addition, Roland Clobus posted his [24th status update of reproducible Debian ISO images](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003327.html). Roland highlights that the images for Debian *unstable* often cannot be generated due to changes in that distribution related to the 64-bit `time_t` transition.

[![]({{ "/images/reports/2024-03/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)

Lastly, Bernhard M. Wiedemann posted another [monthly update](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/NAST5PZPQGJ5JTHYAM6CWB7PCNCLLK6P/) for his reproducibility work in openSUSE.

<br>

### Mailing list highlights

Elsewhere on [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month:

* Alexander Railean of [Siemens](https://www.siemens.com/) asked the list to aid in understanding [how one can independently verify the reproducibility of Java projects](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003311.html) from the [Maven Central](https://central.sonatype.com/) repository. Having explored those repositories, Alexander could not find examples where the `buildinfo` file was present. Arnout Engelen [responded with some details](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003312.html).

* Fay Stegerman [resuscitated a long-dormant thread](https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003278.html) to report that she added support in her [`diff-zip-meta.py` tool](https://github.com/obfusk/reproducible-apk-tools#diff-zip-metapy) to expose extra timestamps embedded in `.zip` and `.apk` metadata.

<br>

### Website updates

[![]({{ "/images/reports/2024-03/website.png#right" | relative_url }})]({{ "/" | relative_url }})

There were made a number of improvements to our website this month, including:

* Pol Dellaiera noticed the frequent need to correctly cite the website itself in academic work. To facilitate easier citation across multiple formats, Pol contributed a [Citation File Format](https://github.com/citation-file-format/citation-file-format) (CIF) file. As a result, an export in [BibTeX](https://www.bibtex.org/) format is now available in the [Academic Publications]({{ "/docs/publications/" | relative_url }}) section. Pol encourages community contributions to further refine the [`CITATION.cff`](https://salsa.debian.org/reproducible-builds/reproducible-website/-/blob/master/CITATION.cff) file. Pol also added an substantial new section to the "[buy in]({{ "/docs/buy-in/" | relative_url }})" page documenting the role of Software Bill of Materials (SBOMs) and ephemeral development environments.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/74e44740)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d9996e2d)]

* Bernhard M. Wiedemann added a new "[commandments]({{ "/docs/commandments/" | relative_url }})" page to the [documentation]({{ "/docs/" | relative_url }})&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4e97c225)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/37e81323)] and fixed some incorrect [YAML](https://yaml.org/) elsewhere on the site&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/569cf016)].

* Chris Lamb add three recent academic papers to the [publications]({{ "/docs/publications/" | relative_url }}) page of the website.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a40c7422)]

* Mattia Rizzolo and Holger Levsen collaborated to add [Infomaniak](https://www.infomaniak.com/) as a sponsor of `amd64` virtual machines.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5d91f1e8)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ac7af0ee)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ba7e9d00)]

* Roland Clobus updated the "[stable outputs]({{ "/docs/stable-outputs/" | relative_url }})" page, dropping version numbers from Python documentation pages&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6fbbb2b3)] and noting that Python's `set` data structure is also affected by the `PYTHONHASHSEED` functionality.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/e9cae80b)]

<br>

### Delta chat clients now reproducible

[![]({{ "/images/reports/2024-03/deltachat.png#right" | relative_url }})](https://chaos.social/@delta/112047758353026678)

[Delta Chat](https://delta.chat), an open source messaging application that can work over email, announced this month that the Rust-based core library underlying Delta chat application [is now reproducible](https://chaos.social/@delta/112047758353026678).

<br>

### [*diffoscope*](https://diffoscope.org)

[![]({{ "/images/reports/2024-03/diffoscope.png#right" | relative_url }})](https://diffoscope.org/)

[diffoscope](https://diffoscope.org) is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions `259`, `260` and `261` to Debian and made the following additional changes:

* New features:

    * Add support for the [`zipdetails`](https://perldoc.perl.org/zipdetails) tool from the Perl distribution. Thanks to Fay Stegerman and Larry Doolittle et al. for the pointer and thread about this tool.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/d9dfe40d)]

* Bug fixes:

    * Don't identify Redis database dumps as [GNU R](https://en.wikipedia.org/wiki/R_(programming_language)) database files based simply on their filename.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/28165345)]
    * Add a missing call to `File.recognizes` so we actually perform the filename check for GNU R data files.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/c6aa6ec8)]
    * Don't crash if we encounter an `.rdb` file without an equivalent `.rdx` file.&nbsp;([#1066991](https://bugs.debian.org/1066991))
    * Correctly check for 7z being available—and not lz4—when testing 7z.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/bd13f8bb)]
    * Prevent a traceback when comparing a contentful `.pyc` file with an empty one.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/c885c24a)]

* Testsuite improvements:

    * Fix `.epub` tests after supporting the new `zipdetails` tool.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/c598dfa7)]
    * Don't use parenthesis within test "skipping…" messages, as PyTest adds its own parenthesis.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f30387bd)]
    * Factor out Python version checking in `test_zip.py`.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/71019a8d)]
    * Skip some Zip-related tests under Python 3.10.14, as a potential regression may have been backported to the 3.10.x series.&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f8270ba8)]
    * Actually test 7z support in the test_7z set of tests, not the lz4 functionality. (Closes: reproducible-builds/diffoscope#359).&nbsp;[[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/529d0ae3)]

In addition, Fay Stegerman updated *diffoscope*'s [monkey patch](https://en.wikipedia.org/wiki/Monkey_patch) for supporting the unusual Mozilla ZIP file format after Python's [`zipfile` module changed to detect potentially insecure overlapping entries within `.zip` files](https://github.com/python/cpython/pull/110016).&nbsp;([#362](https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/362))

Chris Lamb also updated the `trydiffoscope` command line client, dropping a build-dependency on the deprecated `python3-distutils` package to fix Debian bug [#1065988](https://bugs.debian.org/1065988)&nbsp;[[…](https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/d217e92)], taking a moment to also refresh the packaging to the latest Debian standards&nbsp;[[…](https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/f0fcf75)]. Finally, Vagrant Cascadian submitted an update for diffoscope version 260 in [GNU Guix](https://guix.gnu.org/).&nbsp;[[…](https://issues.guix.gnu.org/69656)]

<br>

### Upstream patches

This month, we wrote a large number of patches, including:

* Bernhard M. Wiedemann:

    * [`helm`](https://github.com/helm/helm/issues/12880) (SSL-related build failure)
    * [`java-21-openjdk`](https://bugzilla.opensuse.org/show_bug.cgi?id=1221224) (parallelism)
    * [`libressl`](https://github.com/libressl/portable/issues/1018) (SSL-related build failure)
    * [`nfdump`](https://build.opensuse.org/request/show/1163778) (date issue)
    * [`python-django-q`](https://build.opensuse.org/request/show/1158939) (avoid stuck build)
    * [`python-smart-open`](https://bugzilla.opensuse.org/show_bug.cgi?id=1221663) (fails to build on single-CPU machines)
    * [`python-stdnum`](https://github.com/arthurdejong/python-stdnum/issues/431) (fails to build in 2039)
    * [`python-yarl`](https://build.opensuse.org/request/show/1157151) (regression)
    * [`qemu`](https://bugzilla.opensuse.org/show_bug.cgi?id=1221340) (build failure)
    * [`rabbitmq-java-client`](https://build.opensuse.org/request/show/1155067) (with Fridrich Strba; Maven timestamp issue)
    * [`rmw`](https://github.com/theimpossibleastronaut/rmw/pull/444) (build fails in 2038)
    * [`warewulf`](https://build.opensuse.org/request/show/1162930) (with Egbert Eich; `cpio` modification time and inode issue)
    * [`wxWidgets`](https://github.com/wxWidgets/wxWidgets/issues/24414) (fails to build in 2038)

* Chris Lamb:

    * [#1066042](https://bugs.debian.org/1066042) filed against [`python-quantities`](https://tracker.debian.org/pkg/python-quantities).
    * [#1066083](https://bugs.debian.org/1066083) filed against [`gnome-maps`](https://tracker.debian.org/pkg/gnome-maps).
    * [#1066084](https://bugs.debian.org/1066084) filed against [`tox`](https://tracker.debian.org/pkg/tox).
    * [#1066085](https://bugs.debian.org/1066085) filed against [`q2cli`](https://tracker.debian.org/pkg/q2cli).
    * [#1067098](https://bugs.debian.org/1067098) filed against [`mpl-sphinx-theme`](https://tracker.debian.org/pkg/mpl-sphinx-theme).
    * [#1067099](https://bugs.debian.org/1067099) filed against [`woof-doom`](https://tracker.debian.org/pkg/woof-doom).
    * [#1067100](https://bugs.debian.org/1067100) filed against [`bochs`](https://tracker.debian.org/pkg/bochs).
    * [#1067101](https://bugs.debian.org/1067101) filed against [`storm-lang`](https://tracker.debian.org/pkg/storm-lang).
    * [#1067102](https://bugs.debian.org/1067102) filed against [`librsvg`](https://tracker.debian.org/pkg/librsvg).
    * [#1067218](https://bugs.debian.org/1067218) filed against [`gretl`](https://tracker.debian.org/pkg/gretl).
    * [#1067483](https://bugs.debian.org/1067483) filed against [`postfix`](https://tracker.debian.org/pkg/postfix).
    * [#1067484](https://bugs.debian.org/1067484) filed against [`node-function-bind`](https://tracker.debian.org/pkg/node-function-bind).
    * [#1067485](https://bugs.debian.org/1067485) filed against [`python-pysaml2`](https://tracker.debian.org/pkg/python-pysaml2).
    * [#1067947](https://bugs.debian.org/1067947) filed against [`golang-github-stvp-tempredis`](https://tracker.debian.org/pkg/golang-github-stvp-tempredis).

* James Addison:

    * [#1065124](https://bugs.debian.org/1065124) filed against [`matplotlib`](https://salsa.debian.org/python-team/packages/matplotlib).
    * [#1066014](https://bugs.debian.org/1066014) filed against [`pathos`](https://tracker.debian.org/pkg/pathos).
    * [#1066016](https://bugs.debian.org/1066016) filed against [`rdflib`](https://tracker.debian.org/pkg/rdflib).
    * [#1066017](https://bugs.debian.org/1066017) filed against [`xonsh`](https://tracker.debian.org/pkg/xonsh).
    * [#1066045](https://bugs.debian.org/1066045) filed against [`maven-bundle-plugin`](https://tracker.debian.org/pkg/maven-bundle-plugin). (This patch was then [uploaded by Mattia Rizzollo](https://bugs.debian.org/1066045#35).)

* Jiří Techet:

    * [`geany`](https://github.com/geany/geany/pull/3785) (toolchain-related issue for `glfw`)

Bernhard M. Wiedemann used reproducibility-tooling to detect and fix packages that added changes in their `%check` section, thus failing when built with the `--no-checks` option. Only half of all openSUSE packages were tested so far, but a large number of bugs were filed, including ones against [`caddy`](https://build.opensuse.org/request/show/1163784), [`exiv2`](https://build.opensuse.org/request/show/1155606), [`gnome-disk-utility`](https://build.opensuse.org/request/show/1157126), [`grisbi`](https://build.opensuse.org/request/show/1157756), [`gsl`](https://rb.zq1.de/compare.factory-20240228/diffs/gsl-compare.out), [`itinerary`](https://build.opensuse.org/request/show/1157317), [`kosmindoormap`](https://build.opensuse.org/request/show/1157313), [`libQuotient`](https://build.opensuse.org/request/show/1157763), [`med-tools`](https://build.opensuse.org/request/show/1156899), [`plasma6-disks`](https://build.opensuse.org/request/show/1157315), [`pspp`](https://bugzilla.opensuse.org/show_bug.cgi?id=1221321), [`python-pypuppetdb`](https://build.opensuse.org/request/show/1155640), [`python-urlextract`](https://build.opensuse.org/request/show/1161342), [`rsync`](https://build.opensuse.org/request/show/1157116), [`vagrant-libvirt`](https://build.opensuse.org/request/show/1157910) and [`xsimd`](https://build.opensuse.org/request/show/1157141).

Similarly, Jean-Pierre De Jesus DIAZ employed reproducible builds techniques in order to test a [proposed refactor of the `ath9k-htc-firmware`](https://issues.guix.gnu.org/69476) package. As the change produced bit-for-bit identical binaries to the previously shipped pre-built binaries:

> I don't have the hardware to test this firmware, but the build produces the same hashes for the firmware so it's safe to say that the firmware should keep working.

<br>

### Reproducibility testing framework

[![]({{ "/images/reports/2024-03/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

The Reproducible Builds project operates a comprehensive testing framework running primarily at [*tests.reproducible-builds.org*](https://tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility.

In March, an enormous number of changes were made by Holger Levsen:

* [Debian](https://debian.org/)-related changes:

    * Sleep less after a so-called "404" package state has occurred.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ba9df5b85)]
    * Schedule package builds more often.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ae6271021)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/f909f0353)]
    * Regenerate all our HTML indexes every hour, but only every 12h for the released suites.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/3fc007604)]
    * Create and update *unstable* and *experimental* base systems on `armhf` again.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/abc342ba0)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/f488c2caf)]
    * Don't reschedule so many "depwait" packages due to the current size of the `i386` architecture queue.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/3ae7321bb)]
    * Redefine our scheduling thresholds and amounts.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/052a886b9)]
    * Schedule untested packages with a higher priority, otherwise slow architectures cannot keep up with the *experimental* distribution growing.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/2dea4493d)]
    * Only create the `stats_buildinfo.png` graph once per day.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/57163cee7)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/6e792692d)]
    * Reproducible Debian dashboard: refactoring, update several more static stats only every 12h.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/10a894d08)]
    * Document how to use `systemctl` with new *systemd*-based services.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/141722598)]
    * Temporarily disable `armhf` and `i386` continuous integration tests in order to get some stability back.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/bec1b358f)]
    * Use the `deb.debian.org` CDN everywhere.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/b9e5d80d3)]
    * Remove the *rsyslog* logging facility on *bookworm* systems.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/de6929151)]
    * Add `zst` to the list of packages which are false-positive diskspace issues.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/4df0658d6)]
    * Detect failures to bootstrap Debian base systems.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/a660d7b7f)]

* [Arch Linux](https://archlinux.org/)-related changes:

    * Temporarily disable builds because the *pacman* package manager is broken.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/d3ceee116)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/d271f4936)]
    * Split `reproducible_html_live_status` and split the scheduling timing .&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/fe098a261)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/7d3b8df0f)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/75916ab3a)]
    * Improve handling when database is locked.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/d9eb99846)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/bb69f8d03)]

* Misc changes:

    * Show failed services that require manual cleanup.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ad33f4824)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/46cf49e7f)]
    * Integrate two new [Infomaniak](https://www.infomaniak.com/) nodes.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/5ea2d1f4a)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/772274e4b)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/e55af4ea1)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/5ea8a9875)]
    * Improve IRC notifications for artifacts.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ad5938903)]
    * Run *diffoscope* in different [*systemd* slices](https://www.freedesktop.org/software/systemd/man/latest/systemd.slice.html).&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/a46517dd6)]
    * Run the node health check more often, as it can now repair some issues.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/c0edd5a46)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/f2d6b1ec0)]
    * Also include the string `Bot` in the `userAgent` for Git. (Re: [#929013](https://bugs.debian.org/929013)).&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/285c1de95)]
    * Document increased `tmpfs` size on our OSUOSL nodes.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/4ced9a545)]
    * Disable memory account for the `reproducible_build` service.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/b04767ab2)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/698721665)]
    * Allow 10 times as many open files for the Jenkins service.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/0f283b17a)]
    * Set `OOMPolicy=continue` and `OOMScoreAdjust=-1000` for both the Jenkins and the `reproducible_build` service.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/1c92680fd)]


Mattia Rizzolo also made the following changes:

* [Debian](https://debian.org/)-related changes:

    * Define a [`systemd` slice](https://www.freedesktop.org/software/systemd/man/latest/systemd.slice.html) to group all relevant services.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ec93ea707)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/9e44007a6)]
    * Add a bunch of quotes in scripts to assuage the `shellcheck` tool.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/16517685d)]
    * Add stats on how many packages have been built today so far.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/f8be35075)]
    * Instruct `systemd-run` to handle *diffoscope*'s exit codes specially.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/2913cf26a)]
    * Prefer the `pgrep` tool over grepping the output of `ps`.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ad47ae3aa)]
    * Re-enable a couple of `i386` and `armhf` architecture builders.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/790aea956)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/095a73f81)]
    * Fix some stylistic issues flagged by the Python *flake8* tool.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/094f7cda1)]
    * Cease scheduling Debian *unstable* and *experimental* on the `armhf` architecture due to the `time_t` transition.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/c9027bad6)]
    * Start a few more `i386` & `armhf` workers.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/fcbcc67dd)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/2a4454e6c)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/44e07b0e2)]
    * Temporarly skip `pbuilder` updates in the *unstable* distribution, but only on the `armhf` architecture.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/308338319)]

* Other changes:

    * Perform some large-scale refactoring on how the `systemd` service operates.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/dc5fd4433)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/622278920)]
    * Move the list of workers into a separate file so it's accessible to a number of scripts.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/fea359c67)]
    * Refactor the `powercycle_x86_nodes.py` script to use the new [IONOS](https://www.ionos.co.uk/) API and its new Python bindings.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/e75326235)]
    * Also fix nph-logwatch after the worker changes.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/d96f48648)]
    * Do not install the [`stunnel`](https://www.stunnel.org/) tool anymore, it shouldn't be needed by anything anymore.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/080296849)]
    * Move temporary directories related to [Arch Linux](https://archlinux.org/) into a single directory for clarity.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ad4f23245)]
    * Update the `arm64` architecture host keys.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/842f1e056)]
    * Use a common [Postfix](https://www.postfix.org/) configuration.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/14c62ff18)]

The following changes were also made by:

* Jan-Benedict Glaw:

    * Initial work to clean up a messy [NetBSD](https://www.netbsd.org/)-related script.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/bd35d1bf7)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/8bd707382)]

* Roland Clobus:

    * Show the installer log if the installer fails to build.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/1fad936a1)]
    * Avoid the minus character (i.e. `-`) in a variable in order to allow for tags in [openQA](https://openqa.debian.net/).&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/48221035b)]
    * Update the schedule of Debian live image builds.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/c2e9625bc)]

* Vagrant Cascadian:

    * Maintenance on the `virt*` nodes is completed so bring them back online.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/8fafb3af5)]
    * Use the fully qualified domain name in configuration.&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/68572764e)]

Node maintenance was also performed by Holger Levsen, Mattia Rizzolo&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/7b35add28)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/554d7eb2b)] and Vagrant Cascadian&nbsp;[[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/114758a02)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/4db2f0b58)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ce64261bb)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/cb768d67f)]

<br>

---

If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)

 * Mastodon: [@reproducible_builds@fosstodon.org](https://fosstodon.org/@reproducible_builds)

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)