--- layout: report year: "2023" month: "12" title: "Reproducible Builds in December 2023" draft: false date: 2024-01-11 19:41:04 --- [![]({{ "/images/reports/2023-12/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) **Welcome to the December 2023 report from the [Reproducible Builds](https://reproducible-builds.org) project!** In these reports we outline the most important things that we have been up to over the past month. As a rather rapid recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries ([more]({{ "/#why-does-it-matter" | relative_url }})). --- ### *Reproducible Builds: Increasing the Integrity of Software Supply Chains* awarded IEEE Software "Best Paper" award [![]({{ "/images/reports/2023-12/ieee-paper.jpg#right" | relative_url }})](https://ieeexplore.ieee.org/abstract/document/9403390) In February 2022, [we announced in these reports]({{ "/reports/2023-02/" | relative_url }}) that a paper written by [Chris Lamb](https://chris-lamb.co.uk) and [Stefano Zacchiroli](https://upsilon.cc/~zack/) was now available in the [March/April 2022 issue of IEEE Software](https://ieeexplore.ieee.org/abstract/document/9403390). Titled [*Reproducible Builds: Increasing the Integrity of Software Supply Chains*](https://arxiv.org/abs/2104.06020) ([PDF](https://arxiv.org/pdf/2104.06020)). This month, however, [IEEE Software](https://www.computer.org/csdl/magazine/so) announced [that this paper has won their Best Paper award](https://twitter.com/ieeesoftware/status/1736684911690436868) for 2022.
### Reproducibility to affect package migration policy in Debian [![]({{ "/images/reports/2023-12/releaseteam.png#right" | relative_url }})](https://lists.debian.org/debian-devel-announce/2023/12/msg00003.html) In a post summarising the activities of the [Debian Release Team](https://wiki.debian.org/Teams/ReleaseTeam) at a recent [in-person Debian event in Cambridge, UK](https://wiki.debian.org/DebianEvents/gb/2023/MiniDebConfCambridge), Paul Gevers announced a change to the way packages are "migrated" into the staging area for the next stable Debian release based on its reproducibility status: > The folks from the Reproducibility Project have come a long way since they started working on it 10 years ago, and we believe it's time for the next step in Debian. Several weeks ago, we enabled a migration policy in our migration software that checks for regression in reproducibility. At this moment, that is presented as just for info, but we intend to change that to delays in the not so distant future. We eventually want all packages to be reproducible. To stimulate maintainers to make their packages reproducible now, we'll soon start to apply a bounty [speedup] for reproducible builds, like we've done with passing [autopkgtests](https://people.debian.org/~eriberto/README.package-tests.html) for years. We'll reduce the bounty for successful autopkgtests at that moment in time.
### Speranza: "Usable, privacy-friendly software signing" [![]({{ "/images/reports/2023-12/2305.06463.png#right" | relative_url }})](https://arxiv.org/abs/2305.06463) Kelsey Merrill, Karen Sollins, Santiago Torres-Arias and Zachary Newman have developed a new system called Speranza, which is aimed at reassuring software consumers that the product they are getting has not been tampered with and is coming directly from a source they trust. A [write-up on TechXplore.com](https://techxplore.com/news/2023-12-boosting-faith-authenticity-source-software.html) goes into some more details: > "What we have done," explains Sollins, "is to develop, prove correct, and demonstrate the viability of an approach that allows the [software] maintainers to remain anonymous." Preserving anonymity is obviously important, given that almost everyone—software developers included—value their confidentiality. This new approach, Sollins adds, "simultaneously allows [software] users to have confidence that the maintainers are, in fact, legitimate maintainers and, furthermore, that the code being downloaded is, in fact, the correct code of that maintainer." [[...](https://techxplore.com/news/2023-12-boosting-faith-authenticity-source-software.html)] [The corresponding paper](https://arxiv.org/abs/2305.06463) is published on the [arXiv](https://arxiv.org/) preprint server in various formats, and the announcement has also been [covered in MIT News](https://news.mit.edu/2023/speranza-boosting-faith-authenticity-open-source-software-1211).
### Nondeterministic Git bundles [![]({{ "/images/reports/2023-12/baecher.png#right" | relative_url }})](https://baecher.dev/stdout/reproducible-git-bundles/) [Paul Baecher](https://baecher.dev/) published an interesting blog post on [*Reproducible git bundles*](https://baecher.dev/stdout/reproducible-git-bundles/). For those who are not familiar with them, Git bundles are used for the "offline" transfer of Git objects without an active server sitting on the other side of a network connection. Anyway, Paul wrote about writing a backup system for his entire system, but: > I noticed that a small but fixed subset of [Git] repositories are getting backed up despite having no changes made. That is odd because I would think that repeated bundling of the same repository state should create the exact same bundle. However [it] turns out that for some, repositories bundling is nondeterministic. Paul goes on to to describe his solution, which involves "forcing git to be single threaded makes the output deterministic". The article was [also discussed on Hacker News](https://news.ycombinator.com/item?id=38764452).
### Output from `libxlst` now deterministic [![]({{ "/images/reports/2023-12/libxlst.png#right" | relative_url }})](https://gitlab.gnome.org/GNOME/libxslt/-/commit/82f6cbf8ca61b1f9e00dc04aa3b15d563e7bbc6d) *libxslt* is the [XSLT](https://en.wikipedia.org/wiki/XSLT) C library developed for the [GNOME project](https://www.gnome.org/), where XSLT itself is an XML language to define transformations for XML files. This month, it was revealed that the [result of the `generate-id()` XSLT function is now deterministic across multiple transformations](https://gitlab.gnome.org/GNOME/libxslt/-/blob/d679f4470df2c79443ff54dbc6bd95afaf4cd876/NEWS#L47-48), fixing many issues with reproducible builds. As the [Git commit](https://gitlab.gnome.org/GNOME/libxslt/-/commit/82f6cbf8ca61b1f9e00dc04aa3b15d563e7bbc6d) by Nick Wellnhofer describes: ``` Rework the generate-id() function to return deterministic values. We use a simple incrementing counter and store ids in the 'psvi' member of nodes which was freed up by previous commits. The presence of an id is indicated by a new "source node" flag. This fixes long-standing problems with reproducible builds, see https://bugzilla.gnome.org/show_bug.cgi?id=751621 This also hardens security, as the old implementation leaked the difference between a heap and a global pointer, see https://bugs.chromium.org/p/chromium/issues/detail?id=1356211 The old implementation could also generate the same id for dynamically created nodes which happened to reuse the same memory. Ids for namespace nodes were completely broken. They now use the id of the parent element together with the hex-encoded namespace prefix. ```
### Community updates [![]({{ "/images/reports/2023-12/website.png#right" | relative_url }})]({{ "/" | relative_url }}) There were made a number of improvements to our website, including Chris Lamb fixing the `generate-draft` script to not blow up if the input files have been corrupted today or even in the past [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/40c10ab9)], Holger Levsen updated the [Hamburg 2023 summit]({{ "/events/hamburg2023/" | relative_url }}) to add a link to farewell post [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0a17754a)] & to add a picture of a Post-It note. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d6f3fa6e)], and Pol Dellaiera updated the paragraph about `tar` and the `--clamp-mtime` flag [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/37e7878f)]. On [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month, Bernhard M. Wiedemann posted an interesting summary on some of the reasons [why packages are still not reproducible](https://lists.reproducible-builds.org/pipermail/rb-general/2023-December/003215.html) in 2023. [![]({{ "/images/reports/2023-12/diffoscope.png#right" | relative_url }})](https://diffoscope.org/) [diffoscope](https://diffoscope.org) is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including processing `objdump` symbol comment filter inputs as Python `byte` (and not `str`) instances [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6d788d7d)] and Vagrant Cascadian extended diffoscope support for [GNU Guix](https://guix.gnu.org/) [[...](https://salsa.debian.org/reproducible-builds/diffoscope/-/commit/f1822463eb39ba673b1037e105a5af59fd04262b)] and updated the version in that distribution to [version 253](https://issues.guix.gnu.org/67980) [[...](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=111d010921fea8c803427dc316086434e748e773)].
### "Challenges of Producing Software Bill Of Materials for Java" [![]({{ "/images/reports/2023-12/2303.11102.png#right" | relative_url }})](https://arxiv.org/abs/2303.11102) Musard Balliu, Benoit Baudry, Sofia Bobadilla, Mathias Ekstedt, Martin Monperrus, Javier Ron, Aman Sharma, Gabriel Skoglund, César Soto-Valero and Martin Wittlinger (!) of the [KTH Royal Institute of Technology](https://www.kth.se/en) in Sweden, have published an article in which they: > … deep-dive into 6 tools and the accuracy of the [SBOMs](https://about.gitlab.com/blog/2022/10/25/the-ultimate-guide-to-sboms/) they produce for complex open-source Java projects. Our novel insights reveal some hard challenges regarding the accurate production and usage of software bills of materials. The [paper is available](https://arxiv.org/abs/2303.11102) on [arXiv](https://arxiv.org/).
### Debian Non-Maintainer campaign [![]({{ "/images/reports/2023-12/debian.png#right" | relative_url }})](https://debian.org/) As mentioned in [previous]({{ "/reports/2023-01/" | relative_url }}) [reports]({{ "/reports/2022-12/" | relative_url }}), the Reproducible Builds team within Debian has been organising a series of online and offline sprints in order to clear the huge backlog of reproducible builds patches submitted by performing so-called NMUs ([Non-Maintainer Uploads](https://wiki.debian.org/NonMaintainerUpload)). During December, Vagrant Cascadian performed a number of such uploads, including: * [`crack`](https://tracker.debian.org/crack) [[...](https://browse.dgit.debian.org/crack.git/commit/?id=4b45271101e0f3cf2ca8f5039e487d8931563011)] ([#1021521](https://bugs.debian.org/1021521) & [#1021522](https://bugs.debian.org/1021522)) * [`dustmite`](https://tracker.debian.org/dustmite) [[...](https://browse.dgit.debian.org/dustmite.git/commit/?id=c2ec8a50eb2497ab8ea974ab4f4e5603691c7cd0)] ([#1020878](https://bugs.debian.org/1020878) & [#1020879](https://bugs.debian.org/1020879)) * [`edid-decode`](https://tracker.debian.org/edid-decode) [[...](https://salsa.debian.org/debian/edid-decode/-/commit/c9dcc3bdda30953543cf8ded821efe13f8269fc6)] ([#1020877](https://bugs.debian.org/1020877)) * [`gentoo`](https://tracker.debian.org/gentoo) [[...](https://browse.dgit.debian.org/gentoo.git/commit/?id=6abbef83574c7028fa034a23471863e5107073e2)] ([#1024284](https://bugs.debian.org/1024284)) * [`haskell98-report`](https://tracker.debian.org/haskell98-report) [[...](https://browse.dgit.debian.org/haskell98-report.git/commit/?id=3fa444643941eb7674d8a3fc6adbce447e4d5d55)] ([#1024007](https://bugs.debian.org/1024007)) * [`infinipath-psm`](https://tracker.debian.org/pkg/infinipath-psm) [[...](https://browse.dgit.debian.org/infinipath-psm.git/commit/?id=5fe0454be2b53b268b379a9c4c4e183b62c4397b)] ([#990862](https://bugs.debian.org/990862)) * [`lcm`](https://tracker.debian.org/lcm) [[...](https://salsa.debian.org/debian/lcm/-/commit/3ee1de8caa666836f5b92cb3a664633343b08615)] ([#1024286](https://bugs.debian.org/1024286)) * [`libapache-mod-evasive`](https://tracker.debian.org/libapache-mod-evasive) [[...](https://browse.dgit.debian.org/libapache-mod-evasive.git/commit/?id=7ed970db575818b0742b276d1700faab75398d4b0)] ([#1020800](https://bugs.debian.org/1020800)) * [`libccrtp`](https://tracker.debian.org/libccrtp) [[...](https://browse.dgit.debian.org/libccrtp.git/commit/?id=3fc542e27e9b26b2c46dab2ee51f84306c441a80)] ([#860470](https://bugs.debian.org/860470)) * [`libinput`](https://tracker.debian.org/libinput) [[...](https://browse.dgit.debian.org/libinput.git/commit/?id=1d5d19721b4c215cb08d758b85301e20b2be1af9)] ([#995809](https://bugs.debian.org/995809)) * [`lirc`](https://tracker.debian.org/lirc) [[...](https://browse.dgit.debian.org/lirc.git/commit/?id=0d817194dde2519f6559b1b7a50516c46aac5b5b)] ([#979019](https://bugs.debian.org/979019), [#979023](https://bugs.debian.org/979023) & [#979024](https://bugs.debian.org/979024)) * [`mm-common`](https://tracker.debian.org/mm-common) [[...](https://browse.dgit.debian.org/mm-common.git/commit/?id=dc903218cc0b9bf783494f6d7280ba24e00092c5)] ([#977177](https://bugs.debian.org/977177)) * [`mpl-sphinx-theme`](https://tracker.debian.org/mpl-sphinx-theme) [[...](https://browse.dgit.debian.org/mpl-sphinx-theme.git/commit/?id=fd0cbbd24f534d6a35cab0ee7261b285fb3d8cfb)] ([#1005826](https://bugs.debian.org/1005826)) * [`psi`](https://tracker.debian.org/psi) [[...](https://browse.dgit.debian.org/psi.git/commit/?id=e6f2b1f720dee8658170ce862cb290f019120d88)] ([#1017473](https://bugs.debian.org/1017473)) * [`python-parse-type`](https://tracker.debian.org/python-parse-type) [[...](https://browse.dgit.debian.org/python-parse-type.git/commit/?id=6e17fbda135ce2987457a715e5fbc742796ada1b)] ([#1002671](https://bugs.debian.org/1002671)) * [`ruby-tioga`](https://tracker.debian.org/ruby-tioga) [[...](https://salsa.debian.org/ruby-team/ruby-tioga/-/commit/05c5ae82168cb054844aa2502c9b782976e3a93f)] ([#1005727](https://bugs.debian.org/1005727)) * [`ucspi-proxy`](https://tracker.debian.org/ucspi-proxy) [[...](https://browse.dgit.debian.org/ucspi-proxy.git/commit/?id=3e0e53d6637f8e1e4600c81bf5eac632cc8c5644)] ([#1024125](https://bugs.debian.org/1024125)) * [`ypserv`](https://tracker.debian.org/pkg/ypserv) [[...](https://browse.dgit.debian.org/ypserv.git/commit/?id=9c8c6a058c1c512047aa64a39338b184c0c2be9f)] ([#983138](https://bugs.debian.org/983138)) In addition, Holger Levsen performed three "no-source-change" NMUs in order to address the last packages without `.buildinfo` files in Debian *trixie*, specifically `lorene` (0.0.0~cvs20161116+dfsg-1.1), `maria` (1.3.5-4.2) and `ruby-rinku` (1.7.3-2.1).
### Reproducibility testing framework [![]({{ "/images/reports/2023-12/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/) The Reproducible Builds project operates a comprehensive testing framework (available at [tests.reproducible-builds.org](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In December, a number of changes were made by Holger Levsen: * [Debian](https://debian.org/)-related changes: * Fix matching packages for the [R programming language](https://en.wikipedia.org/wiki/R_(programming_language). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8d62713bf)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b6d7d332)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/37f4c1c08)] * Add a [Certbot](https://certbot.eff.org/) configuration for the Nginx web server. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fa5d87536)] * Enable debugging for the `create-meta-pkgs` tool. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f5ec1b5df)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b38717780)] * [Arch Linux](https://archlinux.org/)-related changes * The `asp` has been deprecated by `pkgctl`; thanks to *dvzrv* for the pointer. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a07e44fa6)] * Disable the Arch Linux builders for now. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fb40264ab)] * Stop referring to the `/trunk` branch / subdirectory. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/22ca4f991)] * Use `--protocol https` when cloning repositories using the `pkgctl` tool. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/42913dcdb)] * Misc changes: * Install the `python3-setuptools` and `swig` packages, which are now needed to build OpenWrt. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/43ceb3a27)] * Install `pkg-config` needed to build Coreboot artifacts. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b0e7b255f)] * Detect failures due to an issue where the [`fakeroot` tool is implicitly required but not automatically installed](https://bugs.debian.org/1058994). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/74f36029a)] * Detect failures due to rename of the `vmlinuz` file. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/deb577757)] * Improve the grammar of an error message. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/64ba2b857)] * Document that `freebsd-jenkins.debian.net` has been updated to [FreeBSD 14.0](https://www.freebsd.org/releases/14.0R/relnotes/). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/516134d7a)] In addition, node maintenance was performed by Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d09194e21)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/708a936ce)].
### Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Bernhard M. Wiedemann: * [`apr`](https://build.opensuse.org/request/show/1133854) (hostname issue) * [`dune`](https://github.com/ocaml/dune/issues/9507) (parallelism) * [`epy`](https://build.opensuse.org/request/show/1134190) (time-based `.pyc` issue) * [`fpc`](https://gitlab.com/freepascal.org/fpc/source/-/issues/40552) (Year 2038) * [`gap`](https://github.com/gap-system/gap/pull/5550) (date) * [`gh`](https://github.com/cli/cli/issues/8452) (FTBFS in 2024) * [`kubernetes`](https://github.com/kubernetes/kubernetes/issues/110928) (fixed random build path) * [`libgda`](https://build.opensuse.org/request/show/1134513) (date) * [`libguestfs`](https://build.opensuse.org/request/show/1133981) (tar) * [`metamail`](https://build.opensuse.org/request/show/1134199) (date) * [`mpi-selector`](https://build.opensuse.org/request/show/1133866) (date) * [`neovim`](https://github.com/neovim/neovim/issues/26387) (randomness in Lua) * [`nml`](https://build.opensuse.org/request/show/1134364) (time-based `.pyc`) * [`pommed`](https://salsa.debian.org/mactel-team/pommed/-/merge_requests/2) (parallelism) * [`procmail`](https://build.opensuse.org/request/show/1130552) (benchmarking) * [`pysnmp`](https://github.com/lextudio/pysnmp/pull/35) (FTBFS in 2038) * [`python-efl`](https://build.opensuse.org/request/show/1130399) (drop Sphinx doctrees) * [`python-pyface`](https://build.opensuse.org/request/show/1131679) (time) * [`python-pytest-salt-factories`](https://build.opensuse.org/request/show/1134187) (time-based `.pyc` issue) * [`python-quimb`](https://bugzilla.opensuse.org/show_bug.cgi?id=1217831) (fails to build on single-CPU systems) * [`python-rdflib`](https://build.opensuse.org/request/show/1132811) (random) * [`python-yarl`](https://build.opensuse.org/request/show/1132599) (random path) * [`qt6-webengine`](https://bugzilla.opensuse.org/show_bug.cgi?id=1217774) (parallelism issue in documentation) * [`texlive`](https://build.opensuse.org/request/show/1133928) (Gzip modification time issue) * [`waf`](https://build.opensuse.org/request/show/1133929) (time-based `.pyc`) * [`warewulf`](https://bugzilla.opensuse.org/show_bug.cgi?id=1217973) (CPIO modification time and inode issue) * [`xemacs`](https://build.opensuse.org/request/show/1134362) (toolchain hostname) * Chris Lamb: * [#1057710](https://bugs.debian.org/1057710) filed against [`python-aiostream`](https://tracker.debian.org/pkg/python-aiostream). * [#1057721](https://bugs.debian.org/1057721) filed against [`openpyxl`](https://tracker.debian.org/pkg/openpyxl). * [#1058681](https://bugs.debian.org/1058681) filed against [`python-multipletau`](https://tracker.debian.org/pkg/python-multipletau). * [#1059013](https://bugs.debian.org/1059013) filed against [`wxmplot`](https://tracker.debian.org/pkg/wxmplot). * [#1059014](https://bugs.debian.org/1059014) filed against [`stunnel4`](https://tracker.debian.org/pkg/stunnel4). * James Addison: * [#1059592](https://bugs.debian.org/1059592) & [#1059631](https://bugs.debian.org/1059631) filed against [`qttools-opensource-src`](https://tracker.debian.org/pkg/qttools-opensource-src).
--- If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * IRC: `#reproducible-builds` on `irc.oftc.net`. * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general) * Mastodon: [@reproducible_builds](https://fosstodon.org/@reproducible_builds) * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)