--- layout: report year: "2023" month: "07" title: "Reproducible Builds in July 2023" draft: false date: 2023-08-04 15:32:47 --- [![]({{ "/images/reports/2023-07/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) Welcome to the July 2023 report from the [Reproducible Builds](https://reproducible-builds.org) project. In our reports, we try to outline the most important things that we have been up to over the past month. As ever, if you are interested in contributing to the project, please visit the [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website. --- [![]({{ "/images/reports/2023-07/H0A2cSejlZ4.jpg#right" | relative_url }})](https://www.youtube.com/watch?v=H0A2cSejlZ4) Marcel Fourné et al. presented at the [IEEE Symposium on Security and Privacy](https://www.ieee-security.org/TC/SP2022/) in San Francisco, CA on [*The Importance and Challenges of Reproducible Builds for Software Supply Chain Security*](https://www.youtube.com/watch?v=H0A2cSejlZ4). As summarised in [last month's report]({{ "/reports/2023-06/" | relative_url }}), the abstract of their paper begins: > The 2020 Solarwinds attack was a tipping point that caused a heightened awareness about the security of the software supply chain and in particular the large amount of trust placed in build systems. Reproducible Builds (R-Bs) provide a strong foundation to build defenses for arbitrary attacks against build systems by ensuring that given the same source code, build environment, and build instructions, bitwise-identical artifacts are created. ([PDF](https://saschafahl.de/static/paper/reprobuilds2023.pdf)) --- [![]({{ "/images/news/supporter-spotlight-simon-butler/simon.jpg#right" | relative_url }})]({{ "/news/2023/08/01/supporter-spotlight-simon-butler/" | relative_url }}) Chris Lamb published an interview with Simon Butler, associate senior lecturer in the School of Informatics at the [University of Skövde](https://www.his.se/en/), on [the business adoption of Reproducible Builds]({{ "/news/2023/08/01/supporter-spotlight-simon-butler/" | relative_url }}). (This is actually the seventh instalment in a series featuring the projects, companies and individuals who support our project. We started this series by [featuring the Civil Infrastructure Platform]({{ "/news/2020/10/21/supporter-spotlight-cip-project/" | relative_url }}) project, and followed this up with a [post about the Ford Foundation]({{ "/news/2021/04/06/supporter-spotlight-ford-foundation/" | relative_url }}) as well as recent ones about [ARDC]({{ "/news/2022/04/14/supporter-spotlight-ardc/" | relative_url }}), the [Google Open Source Security Team (GOSST)]({{ "/news/2022/04/26/supporter-spotlight-google-open-source-security-team/" | relative_url }}), [Bootstrappable Builds]({{ "/news/2022/05/18/jan-nieuwenhuizen-on-bootrappable-builds-gnu-mes-and-gnu-guix/" | relative_url }}), [the F-Droid project]({{ "/news/2022/06/24/supporter-spotlight-hans-christoph-steiner-f-droid-project/" | relative_url }}) and [David A. Wheeler]({{ "/news/2022/12/15/supporter-spotlight-davidawheeler-supply-chain-security/" | relative_url }}).) [![]({{ "/images/reports/2023-07/fossy.png#right" | relative_url }})](https://2023.fossy.us/schedule/presentation/118/) Vagrant Cascadian presented [*Breaking the Chains of Trusting Trust*](https://2023.fossy.us/schedule/presentation/118/) at [FOSSY 2023](https://2023.fossy.us/). --- Rahul Bajaj has been working with Roland Clobus on [merging an overview of environment variations](https://salsa.debian.org/reproducible-builds/reproducible-website/-/merge_requests/97) to [our website]({{ "/" | relative_url }}): > I have identified 16 root causes for unreproducible builds in my empirical study, which I have linked to the corresponding documentation. The initial MR right now contains information about 10 root causes. For each root cause, I have provided a definition, a notable instance, and a workaround. However, I have only found workarounds for 5 out of the 10 root causes listed in this merge request. In the upcoming commits, I plan to add an additional 6 root causes. I kindly request you review the text for any necessary refinements, modifications, or corrections. Additionally, I would appreciate the help with documentation for the solutions/workarounds for the remaining root causes: Archive Metadata, Build ID, File System Ordering, File Permissions, and Snippet Encoding. Your input on the identified root causes for unreproducible builds would be greatly appreciated. [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/-/merge_requests/97)] --- [![]({{ "/images/reports/2023-07/summit.jpg#right" | relative_url }})]({{ "/news/2023/07/05/reproducible-builds-hamburg-meeting/" | relative_url }}) Just a reminder that our [upcoming Reproducible Builds Summit]({{ "/news/2023/07/05/reproducible-builds-hamburg-meeting/" | relative_url }}) is set to take place from **October 31st — November 2nd 2023 in Hamburg, Germany**. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. If you're interested in joining us this year, please make sure to read the [event page]({{ "/news/2023/07/05/reproducible-builds-hamburg-meeting/" | relative_url }}) which has more details about the event and location. --- [![]({{ "/images/reports/2023-07/golang.png#right" | relative_url }})](https://go.dev) There was more progress towards making the [Go programming language](https://go.dev) ecosystem reproducible this month, including: * Adding a new subpage on the GoLang website to show reproduction of the published Go binaries, along with release candidates along with a new binary to produce the given results. This has resulted in [page of unreproducible packages](https://swtch.com/tmp/gorebuild-fail.html) as well as a [reproducible releases](https://swtch.com/tmp/gorebuild-full.html) page. This was achieved via Go bug [#513700](https://go-review.googlesource.com/c/website/+/513700). * In addition [a tool to "reproduce posted Go binaries"](https://go-review.googlesource.com/c/build/+/513975) was added: "This command rebuilds or verifies all the artifacts […] for the latest supported releases. In addition, [*kpcyrd* posted to our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2023-July/003035.html) to report that: > while packaging `govulncheck` for [Arch Linux](https://archlinux.org/) I noticed a checksum mismatch for a tar file I downloaded from `go.googlesource.com`. I used [diffoscope](https://diffoscope.org) to compare the `.tar` file I downloaded with the `.tar` file the build server downloaded, and noticed the timestamps are different. --- [![]({{ "/images/reports/2023-07/debian.png#right" | relative_url }})](https://debian.org/) In Debian, 20 reviews of Debian packages were added, 25 were updated and 25 were removed this month adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). A number of issue types were updated, including marking `ffile_prefix_map_passed_to_clang` being fixed since Debian *bullseye* [[…](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/ab60fa35)] and adding a Debian bug tracker reference for the `nondeterminism_added_by_pyqt5_pyrcc5` issue [[…](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/432a794c)]. In addition, Roland Clobus posted another [detailed update of the status of reproducible Debian ISO images](https://lists.reproducible-builds.org/pipermail/rb-general/2023-July/003021.html) on our mailing list. In particular, Roland helpfully summarised that "live images are looking good, and the number of (passing) automated tests is growing". [![]({{ "/images/reports/2023-07/opensuse.png#right" | relative_url }})](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/MUA7AZODPE6XOCI5AZWVFBDPWD6W5MEN/)
Bernhard M. Wiedemann published another [monthly report about reproducibility within openSUSE](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/MUA7AZODPE6XOCI5AZWVFBDPWD6W5MEN/).
[![]({{ "/images/reports/2023-07/fdroid.png#right" | relative_url }})](https://f-droid.org/en/) F-Droid added 20 new reproducible apps in July, making 165 apps in total that are published with Reproducible Builds and using the upstream developer's signature. [[…](https://gitlab.com/obfusk/fdroid-misc-scripts/-/blob/master/reproducible/overview.md)]
The [Sphinx](https://github.com/sphinx-doc/sphinx) documentation tool recently [accepted a change to improve deterministic reproducibility of documentation](https://github.com/sphinx-doc/sphinx/pull/11312). It's internal `util.inspect.object_description` attempts to sort collections, but this can fail. The change handles the failure case by using string-based object descriptions as a fallback deterministic sort ordering, as well as adding recursive object-description calls for list and tuple datatypes. As a result, documentation generated by Sphinx will be more likely to be automatically reproducible. Lastly in news, *kpcyrd* posted to our [mailing list](https://lists.reproducible-builds.org/pipermail/rb-general) announcing a new "[`repro-env`](https://github.com/kpcyrd/repro-env)" tool: > My initial interest in reproducible builds was "how do I distribute pre-compiled binaries on [GitHub](https://github.com) without people raising security concerns about them". I've cycled back to this original problem about 5 years later and built a tool that is meant to address this. [[…](https://lists.reproducible-builds.org/pipermail/rb-general/2023-July/003026.html)] --- ## Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Bernhard M. Wiedemann: * [`django-graphql-jwt`](https://github.com/flavors/django-graphql-jwt/issues/318) (fails to build in 2038) * [`doxygen`](https://build.opensuse.org/request/show/1099185) (filesystem ordering issue) * [`git-interactive-rebase-tool`](https://github.com/MitMaro/git-interactive-rebase-tool/pull/881) (date-related issue) * [`obs-build`](https://github.com/openSUSE/obs-build/pull/944) * [`procmeter`](https://build.opensuse.org/request/show/1098333) (parallelism race condition) * [`promu`](https://github.com/prometheus/promu/pull/267) * [`python-cx_Freeze`](https://build.opensuse.org/request/show/1100867) (version update for year 2038 fix) * [`python-zope.deprecation`](https://build.opensuse.org/request/show/1100560) * [`python310`](https://bugzilla.opensuse.org/show_bug.cgi?id=1213463) (ASLR-related issue) * [`python-control`](https://github.com/python-control/python-control/issues/927) (fails to build-j4) * [`python-DateTime`](https://github.com/zopefoundation/DateTime/issues/56) (fails to build in 2038) * [`python-pyface`](https://github.com/enthought/pyface/issues/1254) (date/time-related issue) * [`python-quantities`](https://github.com/python-quantities/python-quantities/issues/225) (date/time-related issue) * [`python-scipy`](https://github.com/scipy/scipy/issues/18987) (date/time-related issue) * [`rpmlint`](https://github.com/rpm-software-management/rpmlint/issues/1092) * [`starship`](https://github.com/starship/starship/pull/5352) (filesystem ordering issue) * [`Telethon`](https://github.com/LonamiWebs/Telethon/pull/4163) * [`xindy`](https://sourceforge.net/p/xindy/bugs/65/) (fails to build in 2036) * [`yt`](https://github.com/yt-project/yt/pull/4609) (filesystem ordering issue) * Drop Sphinx doctrees in [`python-bpython`](https://build.opensuse.org/request/show/1100750), [`python-flup`](https://build.opensuse.org/request/show/1101194), [`python-mysqlclient`](https://build.opensuse.org/request/show/1100753), [`python-waitress`](https://build.opensuse.org/request/show/1100756), [`python-WebOb`](https://build.opensuse.org/request/show/1100758), [`python-WebTest`](https://build.opensuse.org/request/show/1100760), [`python-zope.event`](https://build.opensuse.org/request/show/1100761), [`python-zope.hookable`](https://build.opensuse.org/request/show/1100764) & [`python-zope.i18nmessageid`](https://build.opensuse.org/request/show/1100765) * Chris Lamb (*lamby*): * [#1040232](https://bugs.debian.org/1040232) filed against [`dotenv-cli`](https://tracker.debian.org/pkg/dotenv-cli). * [#1040904](https://bugs.debian.org/1040904) filed against [`unity-java`](https://tracker.debian.org/pkg/unity-java). * [#1041840](https://bugs.debian.org/1041840) filed against [`ruby-babosa`](https://tracker.debian.org/pkg/ruby-babosa) ([forwarded upstream](https://github.com/norman/babosa/pull/74)). * [#1041842](https://bugs.debian.org/1041842) filed against [`guidata`](https://tracker.debian.org/pkg/guidata) ([forwarded upstream](https://github.com/Codra-Ingenierie-Informatique/guidata/pull/61)). * Johannes Schauer Marin Rodrigues (*josch*): * [#1040450](https://bugs.debian.org/1040450) filed against [`binutils`](https://tracker.debian.org/pkg/binutils). * John Neffenger: * [openjdk/jfx#446](https://github.com/openjdk/jfx/pull/446) (openjfx), "Enable reproducible builds with `SOURCE_DATE_EPOCH`," a three-and-a-half year effort started by Bernhard M. Wiedemann in January 2020, taken over by John Neffenger in March 2021, integrated upstream in June 2023, and available starting with JavaFX 21 on September 19, 2023. --- [![]({{ "/images/reports/2023-07/diffoscope.png#right" | relative_url }})](https://diffoscope.org) In [*diffoscope*](https://diffoscope.org) development this month, versions `244`, `245` and `246` were uploaded to Debian *unstable* by Chris Lamb, who also made the following changes: * Don't include the file size in image metadata. It is, at best, distracting, and it is already in the directory metadata. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/006f3dc2)] * Add compatibility with `libarchive-5`. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/3c4e378d)] * Mark that the `test_dex::test_javap_14_differences` test requires the `procyon` tool. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/375be212)] * Initial work on DOS/MBR extraction. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/29552993)] * Move to using `assert_diff` in the `.ico` and `.jpeg` tests. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/633bff8c)] * Temporarily mark some Android-related as `XFAIL` due to Debian bugs [#1040941](https://bugs.debian.org/1040941) & [#1040916](https://bugs.debian.org/1040916). [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/5c3c4a7f)] * Fix the "test skipped" reason generation in the case of a version outside of the required range. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2f3dd0d9)] * Update copyright years. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/99f85671)][[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/8b336eb1)] * Fix [*try.diffoscope.org*](https://try.diffoscope.org/). [[…](https://lists.reproducible-builds.org/pipermail/rb-general/2023-July/003033.html)] In addition, Gianfranco Costamagna added support for [LLVM](https://llvm.org/) version 16. [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/80967c67)] --- ## Testing framework [![]({{ "/images/reports/2023-05/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/) The Reproducible Builds project operates a comprehensive testing framework (available at [tests.reproducible-builds.org](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In July, a number of changes were made by Holger Levsen: * General changes: * Upgrade [Jenkins](https://www.jenkins.io/) host to Debian *bookworm* now that Debian 12.1 is out. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/990ac8e38)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a790b6ea9)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/987cc4684)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8db7f6286)] * djm: improve UX when rebooting a node fails. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1289b81d7)] * djm: reduce wait time between rebooting nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f3add0b0b)] * [Debian](https://debian.org/)-related changes: * Various refactoring of the Debian scheduler. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f808fe4b1)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4360d8eb5)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f7bd6aa03)] * Make Debian "live" builds more robust with respect to [*salsa.debian.org*](https://salsa.debian.org/) returning HTTP 502 errors. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7764c040f)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/bfc8513cf)] * Use the legacy SCP protocol instead of the SFTP protocol when transfering Debian "live" builds. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0faadb633)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/334e8b6cc)] * Speed up a number of database queries — thanks, Myon! [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c27c9ddf3)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4153d0d0c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7009e6160)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/db5b3f1ab)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3c55d007f)] * Split `create_meta_pkg_sets` job into two (for Debian *unstable* and Debian *testing*) to half the job runtime to approximately 90 minutes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e14d6d377)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/63190034d)] * Split scheduler job into four separate jobs, one for each tested architecture. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6e84a76e6)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/bf6445062)] * Treat more [PostgreSQL](https://www.postgresql.org/) errors as serious (for some jobs). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b1c14191c)] * Re-enable automatic database documentation now that `postgresql_autodoc` is back in Debian *bookworm*. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6ee50acfa)] * Remove various hardcoding of Debian release names. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/efcdcfe06)] * Drop some i386 special casing. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/81aaf63e3)] * Other distributions: * Speed up [Alpine](https://www.alpinelinux.org/) SQL queries. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/633eabf53)] * Adjust CSS layout for [Arch Linux](https://archlinux.org/) pages to match 3 and not 4 repos being tested. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a5c86800a)] * Drop the 'community' Arch Linux repo as it has now been merged into the 'extra' repo. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/79fb566f9)] * Speed up a number of Arch-related database queries. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0950ab369)] * Try harder to properly cleanup after building [OpenWrt](https://openwrt.org/) packages. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f76fee735)] * Drop all `kfreebsd`-related tests now that it's officially dead. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b987d45d1)] * System health: * Always ignore some well-known harmless orphan processes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/520b9cf44)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/81a961a33)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8d82b5299)] * Detect another case of job failure due to Jenkins shutdown. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5817d34ce)] * Show all non 'co-installable' package sets on the status page. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3c85af5a0)] * Warn that some specific reboot nodes are currently false-positives. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/33befed9f)] * Node health checks: * Run system and node health checks for [Jenkins](https://www.jenkins.io/) less frequently. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a62a0f70e)] * Try to restart any failed `dpkg-db-backup` [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fea6c5d47)] and `munin-node services` [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fd0539eab)]. In addition, Vagrant Cascadian updated the paths in our automated to tests to use the [same paths used by the official Debian build servers](https://bugs.debian.org/1034424). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ef0edadb6)]
--- If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * IRC: `#reproducible-builds` on `irc.oftc.net`. * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general) * Mastodon: [@reproducible_builds@fosstodon.org](https://fosstodon.org/@reproducible_builds) * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)