--- layout: report year: "2023" month: "04" title: "Reproducible Builds in April 2023" draft: false date: 2023-05-06 19:55:17 --- **Welcome to the April 2023 report from the [Reproducible Builds](https://reproducible-builds.org) project!** {: .lead} [![]({{ "/images/reports/2023-04/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) In these reports we outline the most important things that we have been up to over the past month. And, as always, if you are interested in contributing to the project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website. ## General news [![]({{ "/images/reports/2023-04/trisquel.png#right" | relative_url }})](https://blog.josefsson.org/2023/04/10/trisquel-is-42-reproducible/) [Trisquel](https://trisquel.info/) is a fully-free operating system building on the work of [Ubuntu Linux](https://ubuntu.com/). This month, [Simon Josefsson](https://blog.josefsson.org/) published an article on his blog titled [*Trisquel is 42% Reproducible!*](https://blog.josefsson.org/2023/04/10/trisquel-is-42-reproducible/). Simon wrote: > The absolute number may not be impressive, but what I hope is at least a useful contribution is that there actually is a number on how much of [Trisquel](https://trisquel.info/) is reproducible. Hopefully this will inspire others to help improve the actual metric. Simon [wrote another blog post](https://blog.josefsson.org/2023/04/15/sigstore-protects-apt-archives-apt-verify-apt-sigstore/) this month on a new tool to ensure that updates to Linux distribution archive metadata (eg. via `apt-get update`) will only use files that have been recorded in a globally immutable and tamper-resistant ledger. A similar solution exists for [Arch Linux](https://archlinux.org/) (called [`pacman-bintrans`](https://github.com/kpcyrd/pacman-bintrans)) [which was announced in August 2021](https://vulns.xyz/2021/08/monthly-report/) where [an archive of all issued signatures](https://pacman-bintrans.vulns.xyz/) is publically accessible.
[Joachim Breitner](https://www.joachim-breitner.de/) wrote [an in-depth blog post](https://www.joachim-breitner.de/blog/802-More_thoughts_on_a_bootstrappable_GHC) on a bootstrap-capable [GHC](https://www.haskell.org/ghc/), the primary compiler for the Haskell programming language. As a quick background to what this is trying to solve, in order to generate a fully trustworthy compile chain, trustworthy root binaries are needed… and a popular approach to address this problem is called [bootstrappable builds](https://bootstrappable.org/) where the core idea is to address previously-circular build dependencies by creating a new dependency path using simpler prerequisite versions of software. Joachim takes an somewhat recursive approach to the problem for Haskell, leading to the inadvertently humourous question: "Can I turn all of GHC into one module, and compile that?" Elsewhere in the world of bootstrapping, Janneke Nieuwenhuizen and Ludovic Courtès wrote a blog post on the [GNU Guix blog](https://guix.gnu.org/en/blog/) announcing [*The Full-Source Bootstrap*](https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/), specifically: > […] the third reduction of the Guix bootstrap binaries has now been merged in the main branch of Guix! If you run `guix pull` today, you get a package graph of more than 22,000 nodes **rooted in a 357-byte program**—something that had never been achieved, to our knowledge, since the birth of Unix. More info about this change [is available on the post itself](https://guix.gnu.org/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/), including: > The full-source bootstrap was once deemed impossible. Yet, here we are, building the foundations of a GNU/Linux distro entirely from source, a long way towards the ideal that the Guix project has been aiming for from the start. > > There are still some daunting tasks ahead. For example, what about the Linux kernel? The good news is that the bootstrappable community has grown a lot, from two people six years ago there are now around 100 people in the `#bootstrappable` IRC channel.
[Michael Ablassmeier](https://abbbi.github.io/) created a script called [*pypidiff*](https://abbbi.github.io//pypidiff/) as they were looking for a way to track differences between packages published on [PyPI](https://pypi.org/). According to Micahel, *pypidiff* "uses [*diffoscope*](https://diffoscope.org/) to create reports on the published releases and automatically pushes them to a GitHub repository." This can be seen on the [*pypi-diff*](https://github.com/pypi-diff) GitHub page ([example](https://github.com/pypi-diff/20230426/blob/master/D/DAJIN2/0.1.8-0.1.9/diff.md#comparing-dajin2-018srcdajin2coreclusteringscreen_difflocipy--dajin2-019srcdajin2coreclusteringscreen_difflocipy)).
[![]({{ "/images/reports/2023-04/eleutherai.png#right" | relative_url }})](https://www.marktechpost.com/2023/04/09/a-new-ai-research-proposes-pythia-a-suite-of-decoder-only-autoregressive-language-models-ranging-from-70m-to-12b-parameters/) [Eleuther AI](https://www.eleuther.ai/), a non-profit AI research group, recently unveiled [Pythia](https://github.com/EleutherAI/pythia), a collection of 16 [Large Language Model](https://en.wikipedia.org/wiki/Large_language_model) (LLMs) trained on public data in the same order designed specifically to facilitate scientific research. According to a [post on MarkTechPost](https://www.marktechpost.com/2023/04/09/a-new-ai-research-proposes-pythia-a-suite-of-decoder-only-autoregressive-language-models-ranging-from-70m-to-12b-parameters/): > Pythia is the only publicly available model suite that includes models that were trained on the same data in the same order [and] all the corresponding **data and tools to download and replicate the exact training process are publicly released** to facilitate further research. These properties are intended to allow researchers to understand how gender bias (etc.) can affected by training data and model scale.
Back in [February's report]({{ "/reports/2023-02/" | relative_url }}) we reported on a series of changes to the [Sphinx documentation generator](https://www.sphinx-doc.org/) that was initiated after attempts to get the [`alembic`](https://tracker.debian.org/pkg/alembic) Debian package to build reproducibly. Although Chris Lamb was able to identify the source problem and [provided a potential patch that might fix it](https://lists.reproducible-builds.org/pipermail/rb-general/2023-February/002862.html), James Addison has taken the issue in hand, leading to a [large amount of activity](https://github.com/sphinx-doc/sphinx/issues/11198) resulting in a [proposed pull request](https://github.com/sphinx-doc/sphinx/pull/11312) that is waiting to be merged.
[![]({{ "/images/reports/2023-04/fdroid.png#right" | relative_url }})](https://lists.zx2c4.com/pipermail/wireguard/2023-April/008045.html) [WireGuard](https://www.wireguard.com/) is a popular [Virtual Private Network](https://en.wikipedia.org/wiki/Virtual_private_network) (VPN) service that aims to be faster, simpler and leaner than other solutions to create secure connections between computing devices. According to a post on the [WireGuard developer mailing list](https://lists.zx2c4.com/pipermail/wireguard/), the [WireGuard Android app](https://f-droid.org/en/packages/com.wireguard.android/) can now be built reproducibly so that its contents can be publicly verified. According to [the post by Jason A. Donenfeld](https://lists.zx2c4.com/pipermail/wireguard/2023-April/008045.html), "the [F-Droid](https://f-droid.org) project now does this verification by comparing [their build of WireGuard](https://f-droid.org/en/packages/com.wireguard.android/) to the build that the WireGuard project publishes. When they match, the new version becomes available. This is very positive news."
Author and public speaker, [V. M. Brasseur](https://en.wikipedia.org/wiki/VM_Brasseur) published a sample chapter from her upcoming book on "corporate open source strategy" which is the topic of [Software Bill of Materials](https://anonymoushash.vmbrasseur.com/2023/04/24/software-bill-of-materials-sbom) (SBOM): > A software bill of materials (SBOM) is defined as “…a nested inventory for software, a list of ingredients that make up software components.” When you receive a physical delivery of some sort, the bill of materials tells you what’s inside the box. Similarly, when you use software created outside of your organisation, the SBOM tells you what’s inside that software. The SBOM is a file that declares the software supply chain (SSC) for that specific piece of software. [[…](https://anonymoushash.vmbrasseur.com/2023/04/24/software-bill-of-materials-sbom)]
Several distributions noticed recent versions of the Linux Kernel are no longer reproducible because the [BPF Type Format](https://www.kernel.org/doc/html/latest/bpf/btf.html) (BTF) metadata is not generated in a deterministic way. This was discussed on the `#reproducible-builds` IRC channel, but no solution appears to be in sight for now. --- ## Community news On [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month: * Larry Doolittle shared an interesting puzzle with the group where [three bytes in a `.zip` file](https://lists.reproducible-builds.org/pipermail/rb-general/2023-April/002919.html) were different between two builds. * Alexis PM wrote a message as they [had observed a difference between binaries available in the Debian archive and the ones on *tests.reproducible-builds.org*](https://lists.reproducible-builds.org/pipermail/rb-general/2023-April/002931.html). The [thread](https://lists.reproducible-builds.org/pipermail/rb-general/2023-April/thread.html#2931) generated a number of replies, including interesting responses from Vagrant Cascadian [[…](https://lists.reproducible-builds.org/pipermail/rb-general/2023-April/002939.html)] and *kpcyrd* [[…](https://lists.reproducible-builds.org/pipermail/rb-general/2023-April/002934.html)]. [![]({{ "/images/reports/2023-04/foss-north.png#right" | relative_url }})](https://foss-north.se/2023) Holger Levsen gave a talk at [foss-north 2023](https://foss-north.se/2023) in Gothenburg, Sweden on the topic of [*Reproducible Builds, the first ten years*](https://foss-north.se/2023/speakers-and-talks.html#hlevsen). Lastly, there were a number of updates to [our website]({{ "/" | relative_url }}), including: * Chris Lamb attempted a number of ways to try and fix literal `{: .lead}` appearing in the page [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b4e11377)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f55c283b)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2d01c3d8)], made all the *Back to who is involved* links italics [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/762c5a6a)], and corrected the syntax of the `_data/sponsors.yml` file [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ec53c429)]. * Holger Levsen added his recent talk [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0314c0db)], added Simon Josefsson, Mike Perry and Seth Schoen to the contributors page [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ea3966bc)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a909974d)][[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d14f94fe)], reworked the *People* page a little [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/759b1ef0)] [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b312ea5f)], as well as fixed spelling of 'Arch Linux' [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/626a4af0)]. Lastly, Mattia Rizzolo moved some old sponsors to a 'former' section [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/85f22ddb)] and Simon Josefsson added Trisquel GNU/Linux. [[…](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/c8316971)]
--- ## Debian [![]({{ "/images/reports/2023-04/debian.png#right" | relative_url }})](https://debian.org/) * Vagrant Cascadian [reported on the Debian's `build-essential` package set](https://lists.reproducible-builds.org/pipermail/rb-general/2023-May/002961.html), which was "inspired by how close we are to making the Debian `build-essential` set reproducible and how important that set of packages are in general". Vagrant mentioned that: "I have some progress, some hope, and I daresay, some fears…". [[…]](https://lists.reproducible-builds.org/pipermail/rb-general/2023-May/002961.html) * Debian Developer [Cyril Brulebois (*kibi*)](https://mraw.org/) filed a bug against [*snapshot.debian.org*](https://snapshot.debian.org/) after they noticed that "there are many missing `dinstalls`" — that is to say, the snapshot service is not capturing 100% of all of historical states of the Debian archive. This is relevant to reproducibility because without the availability historical versions, it is becomes impossible to repeat a build at a future date in order to correlate checksums. .[…](https://bugs.debian.org/1031628) * 20 reviews of Debian packages were added, 21 were updated and 5 were removed this month adding to our [knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb added a new `build_path_in_line_annotations_added_by_ruby_ragel` toolchain issue. [[…]](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/f62c135d) * Mattia Rizzolo announced that the data for the *stretch* archive on *tests.reproducible-builds.org* [has been archived](https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20230424/014118.html). This matches the [archival of *stretch* within Debian itself](https://lists.debian.org/debian-devel-announce/2023/03/msg00006.html). This is of some historical interest, as *stretch* was the first Debian release regularly tested by the Reproducible Builds project. --- ## Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Bernhard M. Wiedemann: * [`ghc`](https://github.com/opensuse-haskell/ghc-rpm-macros/pull/1) (workaround a parallelism-related issue) * Jan Zerebecki: * [`ghc`](https://gitlab.haskell.org/ghc/ghc/-/issues/23299) (report a parallelism-related issue) * Chris Lamb: * [#1034147](https://bugs.debian.org/1034147) filed against [`ruby-regexp-parser`](https://tracker.debian.org/pkg/ruby-regexp-parser). * Vagrant Cascadian: * [#1033954](https://bugs.debian.org/1033954), [#1033955](https://bugs.debian.org/1033955) and [#1033957](https://bugs.debian.org/1033957) filed against [`pike8.0`](https://tracker.debian.org/pkg/pike8.0). * [#1033958](https://bugs.debian.org/1033958) and [#1033959](https://bugs.debian.org/1033959) filed against [`binutils`](https://tracker.debian.org/pkg/binutils). * [#1034129](https://bugs.debian.org/1034129) filed against [`lomiri-action-api`](https://tracker.debian.org/pkg/lomiri-action-api). * [#1034199](https://bugs.debian.org/1034199) and [#1034200](https://bugs.debian.org/1034200) filed against [`lomiri`](https://tracker.debian.org/pkg/lomiri). * [#1034327](https://bugs.debian.org/1034327) filed against [`nmodl`](https://tracker.debian.org/pkg/nmodl). * [#1034423](https://bugs.debian.org/1034423) filed against [`php8.2`](https://tracker.debian.org/pkg/php8.2). * [#1034431](https://bugs.debian.org/1034431) filed against [`qemu`](https://tracker.debian.org/pkg/qemu). * [#1034499](https://bugs.debian.org/1034499) filed against [`twisted`](https://tracker.debian.org/pkg/twisted). * [#1034740](https://bugs.debian.org/1034740) filed against [`boost1.74`](https://tracker.debian.org/pkg/boost1.74). * [#1034892](https://bugs.debian.org/1034892) filed against [`php8.2`](https://tracker.debian.org/pkg/php8.2). * [#1035324](https://bugs.debian.org/1035324) filed against [`shaderc`](https://tracker.debian.org/pkg/shaderc). * [#1035329](https://bugs.debian.org/1035329) and [#1035331](https://bugs.debian.org/1035331) filed against [`jackd2`](https://tracker.debian.org/pkg/jackd2). --- ## [*diffoscope*](https://diffoscope.org) development [![]({{ "/images/reports/2023-04/diffoscope.png#right" | relative_url }})](https://diffoscope.org) [*diffoscope*](https://diffoscope.org) version `241` was [uploaded to Debian unstable](https://tracker.debian.org/news/1429548/accepted-diffoscope-241-source-into-unstable/) by Chris Lamb. It [included contributions already covered in previous months](https://salsa.debian.org/reproducible-builds/diffoscope/commits/241) as well a change by Chris Lamb to add a missing `raise` statement that was [accidentally dropped in a previous commit](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2d95ae41efad). [[…](https://salsa.debian.org/reproducible-builds/diffoscope/commit/52a55da5)]
--- ## Testing framework [![]({{ "/images/reports/2023-04/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/) The Reproducible Builds project operates a comprehensive testing framework (available at [tests.reproducible-builds.org](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In April, a number of changes were made, including: * Holger Levsen: * Significant work on a new Documented Jenkins Maintenance (djm) script to support logged maintenance of nodes, etc. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/291bc540d)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ddd8e480e)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/63b29e66f)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/2a08b4568)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/ad826396b)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/05d3c235a)] * Add the new APT repo url for Jenkins itself with a new signing key. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/1990ba553)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/f1733a4ba)] * In the Jenkins shell monitor, allow 40 GiB of files for [*diffoscope*](https://diffoscope.org) for the Debian *experimental* distribution as Debian is frozen around the release at the moment. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/7e83620d0)] * Updated Arch Linux testing to cleanup leftover files left in `/tmp/archlinux-ci/` after three days. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/e9cb00e87)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/065d4e172)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/bd09d3dcc)] * Mark a number of nodes hosted by [Oregon State University Open Source Lab](https://osuosl.org/) (OSUOSL) as online and offline. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/7121c81c6)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/bd84a1b6f)][[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/cdd4b5c15)] * Update the node health checks to detect failures to end `schroot` sessions. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/b9f3487dd)] * Filter out another duplicate contributor from the contributor statistics. [[…](https://salsa.debian.org/qa/jenkins.debian.net/commit/462fa2454)] * Mattia Rizzolo: * Code changes to properly [archive the `stretch` Debian distribution](https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20230424/014118.html). [[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/f0f2dab30369c4a44c7bb7a8fe0129dcb9db011c)][[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/1a3aaf3c47df97adeea95016b1e9679a7b1172c2)][[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/cabef2f6ac8defa5b9f809b704a08e4a6ee879dc)][[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/7a90bbd77e5eb2a6aff2ecd97fe32faa7024ab31)] * Introduce the "archived suites" configuration option. [[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/dadd1a38c798058399c1d8aa7dfd10c90853cb47)]][[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/66df6f97098063a41a4f1479ce2425928265957a)] * Fix the [KGB bot](https://salsa.debian.org/kgb-team/kgb/-/wikis/home) configuration to support `pyyaml` 6.0 as present in Debian *bookworm*. [[…](https://salsa.debian.org/qa/jenkins.debian.net/-/commit/bf6fdb5edeb2a57afe25cb9b3cb2f76b9fbf0068)]

--- If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via: * IRC: `#reproducible-builds` on `irc.oftc.net`. * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds) * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)