---
layout: report
year: "2023"
month: "03"
title: "Reproducible Builds in March 2023"
draft: false
date: 2023-04-06 13:52:40
---

**Welcome to the March 2023 report from the [Reproducible Builds](https://reproducible-builds.org) project.**
{: .lead}

[![]({{ "/images/reports/2023-03/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)

In these reports we outline the most important things that we have been up to over the past month. As a quick recap, the motivation behind the reproducible builds effort is to ensure no malicious flaws have been introduced during compilation and distributing processes. It does this by ensuring identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

If you are interested in contributing to the project, please do visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.

<br>

## News

[![]({{ "/images/reports/2023-03/golang.png#right" | relative_url }})](https://github.com/golang/go/issues/57120)

There was progress towards making the [Go programming language](https://go.dev) reproducible this month, with the overall goal remaining making the Go binaries distributed from Google and by [Arch Linux](https://archlinux.org/) (and others) to be bit-for-bit identical. These changes could become part of the upcoming version 1.21 release of Go. An [issue in the Go issue tracker (#57120)](https://github.com/golang/go/issues/57120) is being used to follow and record progress on this.

<br>

Arnout Engelen updated [our website]({{ "/" | relative_url }}) to add and update reproducibility-related links for [NixOS](https://nixos.org/) to [*reproducible.nixos.org*](https://reproducible.nixos.org).&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/04aa8023)]. In addition, Chris Lamb made some cosmetic changes to our [presentations and resources]({{ "/resources/" | relative_url }}) page.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/18424a3c)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/05507724)]

<br>

[![]({{ "/images/reports/2023-03/intel.png#right" | relative_url }})](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html)

[Intel](https://www.intel.com) published [a guide](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html) on how to reproducibly build their Trust Domain Extensions (TDX) firmware. TDX here refers to an Intel technology that combines their existing virtual machine and memory encryption technology with a new kind of virtual machine guest called a Trust Domain. This runs the CPU in a mode that protects the confidentiality of its memory contents and its state from any other software.

<br>

A [reproducibility-related bug from early 2020](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93371) in the [GNU GCC compiler](https://gcc.gnu.org/) as been fixed. The issues was that if GCC was invoked via the `as` frontend, the `-ffile-prefix-map` was being ignored. We were tracking this in Debian via the [`build_path_captured_in_assembly_objects`](https://tests.reproducible-builds.org/debian/issues/unstable/build_path_captured_in_assembly_objects_issue.html) issue. It has now been fixed and will be reflected in GCC version 13.

<br>

[![]({{ "/images/reports/2023-03/foss-north.png#right" | relative_url }})](https://foss-north.se/2023)

Holger Levsen will present at [foss-north 2023](https://foss-north.se/2023) in April of this year in Gothenburg, Sweden on the topic of *Reproducible Builds, the first ten years*.

<br>

Anthony Andreoli, Anis Lounis, Mourad Debbabi and Aiman Hanna of the [Security Research Centre](https://www.concordia.ca/ginacody/research/security-research-centre.html) at [Concordia University, Montreal](https://www.concordia.ca/) published a paper this month entitled [*On the prevalence of software supply chain attacks: Empirical study and investigative framework*](https://www.sciencedirect.com/science/article/abs/pii/S2666281723000094):

> Software Supply Chain Attacks (SSCAs) typically compromise hosts through trusted but infected software. The intent of this paper is twofold: First, we present an empirical study of the most prominent software supply chain attacks and their characteristics. Second, we propose an investigative framework for identifying, expressing, and evaluating characteristic behaviours of newfound attacks for mitigation and future defense purposes. We hypothesize that these behaviours are statistically malicious, existed in the past, and thus could have been thwarted in modernity through their cementation x-years ago.&nbsp;[[...](https://www.sciencedirect.com/science/article/abs/pii/S2666281723000094)]

<br>

On [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month:

* Mattia Rizzolo is asking everyone in the community to save the date for the 2023's Reproducible Builds summit which will take place between October 31st and November 2nd at [Dock Europe](https://dock-europe.net/) in Hamburg, Germany. Separate announcement(s) to follow.&nbsp;[[...](https://lists.reproducible-builds.org/pipermail/rb-general/2023-March/002915.html)]

* *ahojlm* posted an message announcing a new project which is "the first project offering bootstrappable and verifiable builds without any binary seeds." That is to say, a way of providing a verifiable path towards trusted software development platform without relying on pre-provided binary code in order to prevent against various forms of [compiler backdoors](https://en.wikipedia.org/wiki/Backdoor_(computing)#Compiler_backdoors). The [project's homepage](http://rbzfp7h25zcnmxu4wnxhespe64addpopah5ckfpdfyy4qetpziitp5qd.onion/) is hosted on Tor ([mirror](https://www.zq1.de/~bernhard/mirror/rbzfp7h25zcnmxu4wnxhespe64addpopah5ckfpdfyy4qetpziitp5qd.onion/)).

<br>

The [minutes and logs from our March 2023 IRC meeting](http://meetbot.debian.net/reproducible-builds/2023/reproducible-builds.2023-03-28-14.58.html) have been published. In case you missed this one, our next IRC meeting will take place on **Tuesday 25th April at 15:00 UTC** on `#reproducible-builds` on the [OFTC](https://oftc.net) network.

<br>

[![]({{ "/images/reports/2023-03/osuosl.jpg#right" | relative_url }})](http://layer-acht.org/thinking/blog/20230214-i-love-osuosl/)

… and as a Valentines Day present, Holger Levsen wrote on his blog on 14th February to express his thanks to [OSUOSL](https://osuosl.org/) for their continuous support of *reproducible-builds.org*.&nbsp;[[...](http://layer-acht.org/thinking/blog/20230214-i-love-osuosl/)]

<br>

---

## Debian

[![]({{ "/images/reports/2023-03/debian.png#right" | relative_url }})](https://debian.org/)

Vagrant Cascadian developed an easier setup for testing debian packages which uses [*sbuild*](https://wiki.debian.org/sbuild)'s "[unshare mode](https://wiki.debian.org/sbuild#Using_unshare_with_mmdebstrap_.28no_root_needed.29)" along and [*reprotest*](https://salsa.debian.org/reproducible-builds/reprotest), our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences.&nbsp;[[...](https://lists.reproducible-builds.org/pipermail/rb-general/2023-March/002917.html)]

<br>

Over 30 reviews of Debian packages were added, 14 were updated and 7 were removed this month, all adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). A number of issues were updated, including the Holger Levsen updating `build_path_captured_in_assembly_objects` to note that it has been fixed for GCC 13&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/3233540c)] and Vagrant Cascadian added new issues to mark packages where the build path is being captured via the [Rust](https://www.rust-lang.org/) toolchain&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/2691bc83)] as well as new categorisation for where [virtual packages](https://www.debian.org/doc/debian-policy/ch-binary.html#s-virtual-pkg) have nondeterministic versioned dependencies&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/ccaaab5f)].

---

## Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

* Bernhard M. Wiedemann:

    * [`cockpit`](https://build.opensuse.org/request/show/1073723) (gzip mtime)
    * [`crmsh`](https://build.opensuse.org/request/show/1072113) (by mcepl: rewrite to avoid python toolchain issue)
    * [`cx_Freeze`](https://github.com/marcelotduarte/cx_Freeze/pull/1860) (merged, FTBFS-2038)
    * [`golangci-lint`](https://build.opensuse.org/request/show/1073618) (date)
    * [`guestfs-tools`](https://build.opensuse.org/request/show/1072597) (gzip mtime)
    * [`perf`](https://lore.kernel.org/linux-perf-users/20230320201841.1133-1-bwiedemann@suse.de/T/#u) (merged, sort python scandir)
    * [`perl-Date-Calc-XS`](https://build.opensuse.org/request/show/1072531) (FTBFS-2038)
    * [`perl-Date-Calc`](https://build.opensuse.org/request/show/1072525) (FTBFS-2038)
    * [`pw3270`](https://github.com/PerryWerneck/pw3270/pull/50) (merged, date)
    * [`python-dtaidistance`](https://build.opensuse.org/request/show/1068462) (drop unreproducible unnecessary file)
    * [`sonic-pi`](https://bugzilla.opensuse.org/show_bug.cgi?id=1208969) (FTBFS-2038)
    * [`spack`](https://github.com/spack/spack/pull/36064) (parallelism)
    * [`tesseract`](https://github.com/tesseract-ocr/tesseract/issues/4030) (fixed, CPU, -march=native)
* Chris Lamb:

    * [#1032409](https://bugs.debian.org/1032409) filed against [`esda`](https://tracker.debian.org/pkg/esda).
    * [#1032759](https://bugs.debian.org/1032759) filed against [`gle-graphics-manual`](https://tracker.debian.org/pkg/gle-graphics-manual).

* Stefan Brüns:

    * [`transfig/fig2dev`](https://sourceforge.net/p/mcj/fig2dev/ci/fc429e73ff70f3da13434a6787ee2c7be41dfa51/) (also [in openSUSE](https://build.opensuse.org/request/show/1070627) ; date in PDF)

* Vagrant Cascadian:

    * [#1033032](https://bugs.debian.org/1033032) filed against [`buddy`](https://tracker.debian.org/pkg/buddy).
    * [#1033089](https://bugs.debian.org/1033089) filed against [`subread`](https://tracker.debian.org/pkg/subread).
    * [#1033663](https://bugs.debian.org/1033663) filed against [`linux`](https://tracker.debian.org/pkg/linux).

In addition, Vagrant Cascadian [filed a bug with a patch](https://savannah.nongnu.org/bugs/index.php?63944) to ensure [GNU Modula-2](https://www.nongnu.org/gm2/homepage.html) supports the [`SOURCE_DATE_EPOCH`]({{ "/docs/source-date-epoch/" | relative_url }}) environment variable.

---

## Testing framework

[![]({{ "/images/reports/2023-03/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

The Reproducible Builds project operates a comprehensive testing framework (available at [tests.reproducible-builds.org](https://tests.reproducible-builds.org)) in order to check packages and other artifacts for reproducibility. In March, the following changes were made by Holger Levsen:

* [Arch Linux](https://archlinux.org/)-related changes:

    * Build Arch packages in `/tmp/archlinux-ci/$SRCPACKAGE` instead of `/tmp/$SRCPACKAGE`.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/83d7c78f0)]
    * Start 2/3 of the builds on the `o1` node, the rest on `o2`.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b798a3111)]
    * Add graphs for Arch Linux (and OpenWrt) builds.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/93c0abe7d)]
    * Toggle Arch-related builders to debug why a specific node overloaded.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2f3214451)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0c68f8d80)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f65123374)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9aaa1a5f9)]

* Node health checks:

    * Detect `SetuptoolsDeprecationWarning` tracebacks in Python builds.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/950256a89)]
    * Detect failures do perform `chdist` calls.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b87374d28)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/39ade4464)]

* [OSUOSL](https://osuosl.org/) node migration.

    * Install `megacli` packages that are needed for hardware RAID.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/649a70080)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/52fae771e)]
    * Add health check and maintenance jobs for new nodes.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/11a38325e)]
    * Add mail config for new nodes.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3ca9e3a8d)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/72f3fbb89)]
    * Handle a node running in the future correctly.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8cdfe61f0)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3bc4af860)]
    * Migrate some nodes to Debian *bookworm*.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/95e7d0773)]
    * Fix nodes health overview for osuosl3.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e29233697)]
    * Make sure the `/srv/workspace` directory is owned by by the `jenkins` user.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c30f8722f)]
    * Use `.debian.net` names everywhere, except when communicating with the outside world.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e3df5ea5d)]
    * Grant *fpierret* access to a new node.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/13c01e232)]
    * Update documentation.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b350fd3d2)]
    * Misc migration changes.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/66051839c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e95106f82)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a8815d1c9)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a9ef6c0c2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/11fc197ac)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9a74cdaf9)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5c9a894a1)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/627e56fa3)]

* Misc changes:

    * Enable fail2ban everywhere and monitor it with munin&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2d5c765d0)].
    * Gracefully deal with non-existing [Alpine](https://www.alpinelinux.org/) schroots.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a8d44f889)]

In addition, Roland Clobus is continuing his work on [reproducible Debian ISO images](https://lists.reproducible-builds.org/pipermail/rb-general/2023-February/002877.html):

* Add/update [openQA](http://open.qa/) configuration&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/962a721f4)], and use the actual timestamp for openQA builds&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f0ac16d13)].
* Moved adding the user to the `docker` group from the `janitor_setup_worker` script to the (more general) `update_jdn.sh` script.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/25835d550)]
* Use the (short-term) 'reproducible' source when generating `live-build` images.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b1c218dad)]

---

## [*diffoscope*](https://diffoscope.org) development

[![]({{ "/images/reports/2023-03/diffoscope.png#right" | relative_url }})](https://diffoscope.org)

[*diffoscope*](https://diffoscope.org) is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats as well. This month, Mattia Rizzolo released versions [`238`](https://diffoscope.org/news/diffoscope-238-released/), and Chris Lamb released versions [`239`](https://diffoscope.org/news/diffoscope-239-released/) and [`240`](https://diffoscope.org/news/diffoscope-240-released/). Chris Lamb also made the following changes:

* Fix compatibility with PyPDF 3.x, and correctly restore test data.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/a91f8bfe)]
* Rework PDF annotation handling into a separate method.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2b268980)]

In addition, Holger Levsen performed a long-overdue overhaul of the [Lintian](https://lintian.debian.org/) overrides in the Debian packaging&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/7d0fd9c3)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/8b0fe07a)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/e430e268)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/04f2114f)], and Mattia Rizzolo updated the packaging to silence an `include_package_data=True`&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f2f30420)], fixed the build under Debian *bullseye*&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/9e525a8e)], fixed tool name in a list of tools permitted to be absent during package build tests&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/36854d06)] and as well as documented sending out an email upon &nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/635c404d)].

In addition, Vagrant Cascadian updated the version of [GNU Guix](https://guix.gnu.org/) to 238&nbsp;[[...](https://issues.guix.gnu.org/62312) and 239&nbsp;[[...](https://issues.guix.gnu.org/62312)]. Vagrant also updated *reprotest* to version 0.7.23.&nbsp;[[...](https://issues.guix.gnu.org/62172)]

---

## Other development work

[![]({{ "/images/reports/2023-03/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)

Bernhard M. Wiedemann published another [monthly report about reproducibility within openSUSE](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/WB5NTG4Q6KK2YLIZ4OF65MI5CQDOEA3B/)

<br>
<br>

---


If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)