--- layout: report year: "2024" month: "02" title: "Reproducible Builds in February 2024" draft: false date: 2024-03-09 16:53:13 --- [![]({{ "/images/reports/2024-02/reproducible-builds.png#right" | relative_url }})]({{ "/" | relative_url }}) **Welcome to the February 2024 report from the [Reproducible Builds](https://reproducible-builds.org) project!** In our reports, we try to outline what we have been up to over the past month as well as mentioning some of the important things happening in software supply-chain security. --- ### [Reproducible Builds at FOSDEM 2024]({{ "/news/2024/02/08/reproducible-builds-at-fosdem-2024/" | relative_url }}) [![]({{ "/images/reports/2024-02/fosdem.jpeg#right" | relative_url }})]({{ "/news/2024/02/08/reproducible-builds-at-fosdem-2024/" | relative_url }}) Core Reproducible Builds developer Holger Levsen presented at the main track at [FOSDEM](https://fosdem.org/2024/) on Saturday 3rd February this year in Brussels, Belgium. However, that wasn't the only talk related to Reproducible Builds. However, please see our [**comprehensive FOSDEM 2024 news post**]({{ "/news/2024/02/08/reproducible-builds-at-fosdem-2024/" | relative_url }}) for the full details and links.
### [*Maintainer Perspectives on Open Source Software Security*](https://www.linuxfoundation.org/research/maintainer-perspectives-on-security?hsLang=en) [![]({{ "/images/reports/2024-02/maintainer-perspectives.png#right" | relative_url }})](https://www.linuxfoundation.org/research/maintainer-perspectives-on-security?hsLang=en) Bernhard M. Wiedemann spotted that a recent report entitled [*Maintainer Perspectives on Open Source Software Security*](https://www.linuxfoundation.org/research/maintainer-perspectives-on-security?hsLang=en) written by Stephen Hendrick and Ashwin Ramaswami of the [Linux Foundation](https://www.linuxfoundation.org/) sports an infographic which mentions that "[56% of [polled] projects support reproducible builds](https://www.linuxfoundation.org/hubfs/LF%20Research/MaintainerSecurityBPs_Infographic.pdf)".
### Three new reproducibility-related academic papers A total of three separate scholarly papers related to Reproducible Builds have appeared this month: [![]({{ "/images/reports/2024-02/arXiv-2401.14635.png#right" | relative_url }})](https://arxiv.org/abs/2401.14635) [*Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors*](https://arxiv.org/abs/2401.14635) by Taylor R. Schorlemmer, Kelechi G. Kalu, Luke Chigges, Kyung Myung Ko, Eman Abdul-Muhd, Abu Ishgair, Saurabh Bagchi, Santiago Torres-Arias and James C. Davis ([Purdue University](https://www.purdue.edu/), Indiana, USA) is concerned with the problem that: > Package maintainers can guarantee package authorship through software signing [but] it is unclear how common this practice is, and whether the resulting signatures are created properly. Prior work has provided raw data on signing practices, but measured single platforms, did not consider time, and did not provide insight on factors that may influence signing. We lack a comprehensive, multi-platform understanding of signing adoption and relevant factors. This study addresses this gap. ([arXiv](https://arxiv.org/abs/2401.14635), [full PDF](https://arxiv.org/pdf/2401.14635.pdf))
[![]({{ "/images/reports/2024-02/arXiv-2402.00424.png#right" | relative_url }})](https://arxiv.org/abs/2402.00424) [*Reproducibility of Build Environments through Space and Time*](https://arxiv.org/abs/2402.00424) by Julien Malka, Stefano Zacchiroli and Théo Zimmermann ([Institut Polytechnique de Paris](https://www.ip-paris.fr/), France) addresses: > [The] principle of reusability […] makes it harder to reproduce projects’ build environments, even though reproducibility of build environments is essential for collaboration, maintenance and component lifetime. In this work, we argue that functional package managers provide the tooling to make build environments reproducible in space and time, and we produce a preliminary evaluation to justify this claim. The abstract continues with the claim that "Using historical data, we show that we are able to reproduce build environments of about 7 million [Nix](https://nixos.org/) packages, and to rebuild 99.94% of the 14 thousand packages from a 6-year-old Nixpkgs revision. ([arXiv](https://arxiv.org/abs/2402.00424), [full PDF](https://arxiv.org/pdf/2402.00424.pdf))
[![]({{ "/images/reports/2024-02/msr24.png#right" | relative_url }})](https://inria.hal.science/hal-04441579v2) [*Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems*](https://inria.hal.science/hal-04441579v2) by Georges Aaron Randrianaina, Djamel Eddine Khelladi, Olivier Zendra and Mathieu Acher ([Inria centre at Rennes University](https://www.inria.fr/en/inria-centre-rennes-university), France): > This paper thus proposes an approach to automatically identify configuration options causing non-reproducibility of builds. It begins by building a set of builds in order to detect non-reproducible ones through binary comparison. We then develop automated techniques that combine statistical learning with symbolic reasoning to analyze over 20,000 configuration options. Our methods are designed to both detect options causing non-reproducibility, and remedy non-reproducible configurations, two tasks that are challenging and costly to perform manually. ([HAL Portal](https://inria.hal.science/hal-04441579v2), [full PDF](https://inria.hal.science/hal-04441579/file/msr24.pdf))
### Mailing list highlights From [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month: * User *cen* posted a query asking "[How to verify a package by rebuilding it locally on Debian](https://lists.reproducible-builds.org/pipermail/rb-general/2024-February/003238.html)" which [received a followup from Vagrant Cascadian](https://lists.reproducible-builds.org/pipermail/rb-general/2024-February/003240.html). * James Addison asked "[Two questions about build-path reproducibility in Debian](https://lists.reproducible-builds.org/pipermail/rb-general/2024-February/003246.html)" regarding the differences in the testing performed by [Debian's GitLab continuous integration (CI) pipeline](https://salsa.debian.org/salsa-ci-team/pipeline) and the [Debian-specific testing performed by the Reproducible Builds project itself](https://tests.reproducible-builds.org/debian/reproducible.html), and followed this with a separate but related question regarding misconfigured [*reprotest*](https://salsa.debian.org/reproducible-builds/reprotest) configurations.
### Distribution work [![]({{ "/images/reports/2024-02/debian.png#right" | relative_url }})](https://debian.org/) In Debian this month, 5 reviews of Debian packages were added, 22 were updated and 8 were removed this month adding to [Debian's knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). A number of issue types were updated as well. [[…]](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/bcae685e)[[…]](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/a3137bef)[[…]](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/6ac62ef7)[[…]](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/c272b790) In addition, Roland Clobus posted his 23rd [update of the status of reproducible ISO images](https://lists.reproducible-builds.org/pipermail/rb-general/2024-February/003251.html) on our mailing list. In particular, Roland helpfully summarised that "all major desktops build reproducibly with *bullseye*, *bookworm*, *trixie* and *sid* provided they are built for a second time within the same DAK run (i.e. [within] 6 hours)" and that there will likely be further work at a [MiniDebCamp in Hamburg](https://wiki.debian.org/DebianEvents/de/2024/MiniDebCampHamburg). Furthermore, Roland also [responded in-depth](https://lists.reproducible-builds.org/pipermail/rb-general/2024-February/003233.html) to a query about a [previous report](https://lists.reproducible-builds.org/pipermail/rb-general/2024-January/003217.html)
[![]({{ "/images/reports/2024-02/fedora.png#right" | relative_url }})](https://github.com/keszybz/fedora-repro-build) [Fedora](https://fedoraproject.org/) developer [Zbigniew Jędrzejewski-Szmek](https://github.com/keszybz) announced a work-in-progress script called [`fedora-repro-build`](https://github.com/keszybz/fedora-repro-build) that attempts to reproduce an existing package within a [koji](https://pagure.io/koji/) build environment. Although the [projects' `README` file](https://github.com/keszybz/fedora-repro-build#readme) lists a number of "fields will always or almost always vary" and there is a non-zero [list of other known issues](https://pagure.io/fedora-reproducible-builds/project/issues?tags=irreproducibility), this is an excellent first step towards full Fedora reproducibility.
[![]({{ "/images/reports/2024-02/archlinux.png#right" | relative_url }})](https://archlinux.org/) Jelle van der Waa [introduced a new linter rule](https://gitlab.archlinux.org/pacman/namcap/-/merge_requests/64) for [Arch Linux](https://archlinux.org/) packages in order to detect cache files leftover by the [Sphinx documentation generator](https://www.sphinx-doc.org/en/master/) which are unreproducible by nature and should not be packaged. At the time of writing, 7 packages in the Arch repository are affected by this.
[![]({{ "/images/reports/2024-02/opensuse.png#right" | relative_url }})](https://www.opensuse.org/) Elsewhere, Bernhard M. Wiedemann posted another [monthly update](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/I66U56F5R3TR4ZTLYGPSGWINNOLZ7XP4/) for his work elsewhere in openSUSE.
### [*diffoscope*](https://diffoscope.org) [![]({{ "/images/reports/2024-02/diffoscope.png#right" | relative_url }})](https://diffoscope.org/) [diffoscope](https://diffoscope.org) is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions `256`, `257` and `258` to Debian and made the following additional changes: * Use a deterministic name instead of trusting `gpg`'s --use-embedded-filenames. Many thanks to Daniel Kahn Gillmor for reporting this issue and providing feedback. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/458f7f04)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/18d69030)] * Don't error-out with a traceback if we encounter `struct.unpack`-related errors when parsing Python `.pyc` files. ([#1064973](https://bugs.debian.org/1064973)). [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/466523ac)] * Don't try and compare `rdb_expected_diff` on non-GNU systems as `%p` formatting can vary, especially with respect to MacOS. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/c09d0a9e)] * Fix compatibility with [`pytest`](https://docs.pytest.org/en/8.0.x/) 8.0. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/ce04e0dd)] * Temporarily fix support for Python 3.11.8. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/5e6cfbf0)] * Use the `7zip` package (over `p7zip-full`) after a Debian package transition. ([#1063559](https://bugs.debian.org/1063559)). [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/43ee3684)] * Bump the minimum [Black source code reformatter](https://black.readthedocs.io/en/stable/) requirement to 24.1.1+. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/00418fb4)] * Expand an older changelog entry with a CVE reference. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/86645633)] * Make `test_zip` black clean. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/10c0c6fc)] In addition, James Addison contributed a patch to parse the headers from the `diff(1)` correctly [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/4648dcfa)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/fa73fc2b)] — thanks! And lastly, Vagrant Cascadian pushed updates in [GNU Guix](https://guix.gnu.org/) for diffoscope to version [255](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=9d52585ebd4d759607eacfef31144676b08edc81), [256](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=30196aec07dab8cc0f4a614b160f1857377a6a84), and [258](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=16ab67182bc1e5b046caee9a2e38b71159703f34), and updated *trydiffoscope* to [67.0.6](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f45d05133472a9da13eae20ba4a676c696682c90).
### [*reprotest*](https://salsa.debian.org/reproducible-builds/reprotest) [*reprotest*](https://salsa.debian.org/reproducible-builds/reprotest) is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, Vagrant Cascadian made a number of changes, including: * Create a (working) proof of concept for enabling a specific number of CPUs. [[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/cab6270)][[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/9d0562d)] * Consistently use 398 days for time variation rather than choosing randomly and update `README.rst` to match. [[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/86365b5)][[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/57ab249)] * Support a new `--vary=build_path.path` option. [[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/f94904b)][[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/9ea2e4b)][[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/9b0f5dc)][[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/94e66c4)]
### Website updates [![]({{ "/images/reports/2024-02/website.png#right" | relative_url }})]({{ "/" | relative_url }}) There were made a number of improvements to our website this month, including: * Chris Lamb: * Improve the relative sizing of headers. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3243e14b)] * Re-order and "punch" up the introduction and documentation on the [`SOURCE_DATE_EPOCH`]({{ "/docs/source-date-epoch/" | relative_url }}) page. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/05a76405)] * Update [`SOURCE_DATE_EPOCH`]({{ "/docs/source-date-epoch/" | relative_url }}) documentation re. `datetime.datetime.fromtimestamp`. Thanks, James Addison. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/502769f1)] * Add a [post about Reproducible Builds at FOSDEM 2024]({{ "/news/2024/02/08/reproducible-builds-at-fosdem-2024/" | relative_url }}). [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b09d3c22)] * Holger Levsen: * Update the [GNU Guix]({{ "/projects/guix" | relative_url }}) page to include their [reproducibility QA page](https://qa.guix.gnu.org/reproducible-builds). [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d33582dc)] * Add Sune Vuorela and Jan-Benedict Glaw to our contributors list. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3bed935a)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8bf556b5)] * Mattia Rizzolo: * Add [Sovereign Tech Fund](https://www.sovereigntechfund.de/)'s logo to our sponsors. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a54f6e20)] * Update our sponsors list. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/de187090)]
### Reproducibility testing framework [![]({{ "/images/reports/2024-02/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/) The Reproducible Builds project operates a comprehensive testing framework (available at [*tests.reproducible-builds.org*](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In February, a number of changes were made by Holger Levsen: * [Debian](https://debian.org/)-related changes: * Temporarily disable upgrading/bootstrapping Debian *unstable* and *experimental* as they are currently broken. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ef88cc3ae)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7ed553444)] * Use the 64-bit `amd64` kernel on all `i386` nodes; no more 686 [PAE](https://en.wikipedia.org/wiki/Physical_Address_Extension) kernels. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/53c3c39bd)] * Add an [Erlang](https://www.erlang.org/) package set. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d29d41e3b)] * Other changes: * Grant Jan-Benedict Glaw shell access to the Jenkins node. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/252598e99)] * Enable debugging for [NetBSD](https://www.netbsd.org/) reproducibility testing. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/091fa73f1)] * Use `/usr/bin/du --apparent-size` in the Jenkins shell monitor. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fd54c037d)] * Revert "reproducible nodes: mark osuosl2 as down". [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/37cc03eef)] * Thanks again to [Codethink](https://www.codethink.co.uk/), for they have doubled the RAM on our `arm64` nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/640c38126)] * Only set `/proc/$pid/oom_score_adj` to -1000 if it has not already been done. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c99da2ef3)] * Add the `opemwrt-target-tegra` and `jtx` task to the list of zombie jobs. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e3b188dff)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7fbed0735)] Vagrant Cascadian also made the following changes: * Overhaul the handling of [OpenSSH](https://www.openssh.com/) configuration files after updating from Debian *bookworm*. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3e58ee08c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7d8a99cb5)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5484a9db0)] * Add two new `armhf` architecture build nodes, `virt32z` and `virt64z`, and insert them into the [Munin monitoring](https://munin-monitoring.org/). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8700924ae)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2c462cc3c)] [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7feece465)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6159ad4f9)] In addition, Alexander Couzens updated the [OpenWrt](https://openwrt.org/) configuration in order to replace the `tegra` target with `mpc85xx` [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b5b63be56)], Jan-Benedict Glaw updated the [NetBSD](https://www.netbsd.org/) build script to use a separate `$TMPDIR` to mitigate out of space issues on a [tmpfs](https://en.wikipedia.org/wiki/Tmpfs)-backed `/tmp` [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/910b83f88)] and Zheng Junjie added a link to the [GNU Guix](https://guix.gnu.org/) tests [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/57b21155e)]. Lastly, node maintenance was performed by Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/01ecc9495)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2f650ed98)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/20e9e5c64)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9ce43116c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9a37e768d)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b7417a2f8)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a2315e19f)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/aa7579a92)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c78087b27)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b9d95648)].
### Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Philip Rinn: * [`gimagereader`](https://github.com/manisandro/gImageReader/pull/667) (date) * Bernhard M. Wiedemann: * [`grass`](https://github.com/OSGeo/grass/pull/3417) (date-related issue) * [`grub2`](https://build.opensuse.org/request/show/1144993) (filesystem ordering issue) * [`latex2html`](https://build.opensuse.org/request/show/1150775) (drop a non-deterministic log) * [`mhvtl`](https://github.com/markh794/mhvtl/pull/128) (tar) * [`obs`](https://github.com/openSUSE/obs-build/issues/980) (build-tool issue) * [`ollama`](https://github.com/ollama/ollama/pull/2836) (GZip embedding the modification time) * [`presenterm`](https://github.com/mfontanini/presenterm/pull/202) (filesystem-ordering issue) * [`qt6-quick3d`](https://bugreports.qt.io/browse/QTBUG-122722) (parallelism) * Chris Lamb: * [#1064506](https://bugs.debian.org/1064506) filed against [`geophar`](https://tracker.debian.org/pkg/geophar). * [#1064891](https://bugs.debian.org/1064891) filed against [`pytest-repeat`](https://tracker.debian.org/pkg/pytest-repeat). * [#1064892](https://bugs.debian.org/1064892) filed against [`klepto`](https://tracker.debian.org/pkg/klepto). * James Addison: * [#1064519](https://bugs.debian.org/1064519) filed against [`flask-limiter`](https://tracker.debian.org/pkg/flask-limiter). * [`python-parsl-doc`](https://bugs.debian.org/1063542) (disable dynamic argument evaluation by Sphinx `autodoc` extension) * [`python3-pytest-repeat`](https://bugs.debian.org/1064891) (remove `entry_points.txt` creation that varied by shell) * [`python3-selinux`](https://bugs.debian.org/1064894) (remove packaged `direct_url.json` file that embeds build path) * [`python3-sepolicy`](https://bugs.debian.org/1064895) (remove packaged `direct_url.json` file that embeds build path) * [#1064575](https://bugs.debian.org/1064575) filed against [`pyswarms`](https://tracker.debian.org/pkg/pyswarms). * [#1064638](https://bugs.debian.org/1064638) filed against [`python-x2go`](https://tracker.debian.org/pkg/python-x2go). * [`snapd`](https://bugs.debian.org/1064404) (fix timestamp header in packaged manual-page) * [`zzzeeksphinx`](https://bugs.debian.org/1042955) (existing RB patch forwarded and merged (with modifications)) * Johannes Schauer Marin Rodrigues: * [#1063939](https://bugs.debian.org/1063939) filed against [`fop`](https://tracker.debian.org/pkg/fop).
--- If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * IRC: `#reproducible-builds` on `irc.oftc.net`. * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds) * Mastodon: [@reproducible_builds@fosstodon.org](https://fosstodon.org/@reproducible_builds) * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)