---
layout: report
year: "2022"
month: "02"
title: "Reproducible Builds in February 2022"
draft: false
date: 2022-03-05 11:17:40
---

[![]({{ "/images/reports/2022-02/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)

**Welcome to the February 2022 report from the [Reproducible Builds](https://reproducible-builds.org) project**. In these reports, we try to round-up the important things we and others have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.

---

[![]({{ "/images/reports/2022-02/towards-build-reproducibility.png#right" | relative_url }})](https://arxiv.org/abs/2202.05906)

Jiawen Xiong, Yong Shi, Boyuan Chen, Filipe R. Cogo and Zhen Ming Jiang have published a new paper titled [*Towards Build Verifiability for Java-based Systems*](https://arxiv.org/abs/2202.05906) ([PDF](https://arxiv.org/pdf/2202.05906.pdf)). The abstract of the paper contains the following:

> Various efforts towards build verifiability have been made to C/C++-based systems, yet the techniques for Java-based systems are not systematic and are often specific to a particular build tool (eg. Maven). In this study, we present a systematic approach towards build verifiability on Java-based systems.

---

[![]({{ "/images/reports/2022-02/gitbom.png#right" | relative_url }})](https://gitbom.dev)

[GitBOM](https://gitbom.dev/) is a flexible scheme to track the source code used to generate build artifacts via [Git-like unique identifiers](https://gitbom.dev/glossary/gitbom/#gitbom-identifier). Although the project has been active for a while, the community around GitBOM has now started running [weekly community meetings](https://gitbom.dev/community/).

---

[![]({{ "/images/reports/2022-02/ieee-paper.jpg#right" | relative_url }})](https://ieeexplore.ieee.org/abstract/document/9403390)

The paper [Chris Lamb](https://chris-lamb.co.uk) and [Stefano Zacchiroli](https://upsilon.cc/~zack/) is now available in the [March/April 2022 issue of IEEE Software](https://ieeexplore.ieee.org/abstract/document/9403390). Titled [*Reproducible Builds: Increasing the Integrity of Software Supply Chains*](https://arxiv.org/abs/2104.06020) ([PDF](https://arxiv.org/pdf/2104.06020)), the abstract of the paper contains the following:

> We first define the problem, and then provide insight into the challenges of making real-world software build in a "reproducible" manner-this is, when every build generates bit-for-bit identical results. Through the experience of the Reproducible Builds project making the Debian Linux distribution reproducible, we also describe the affinity between reproducibility and quality assurance (QA).

---

[![]({{ "/images/reports/2022-02/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)

In [openSUSE](https://www.opensuse.org/), Bernhard M. Wiedemann posted his [monthly reproducible builds status report](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/Y5VPNAVHSXNTUP2T6XXK7MZGJO24JONF/).

---

On [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month, Thomas Schmitt started a thread around the [`SOURCE_DATE_EPOCH` specification](https://reproducible-builds.org/specs/source-date-epoch/) related to formats that cannot help [embedding potentially timezone-specific timestamp](https://lists.reproducible-builds.org/pipermail/rb-general/2022-February/002483.html). ([Full thread index](https://lists.reproducible-builds.org/pipermail/rb-general/2022-February/thread.html#2483).)

---

[![]({{ "/images/reports/2022-02/yoctoproject.png#right" | relative_url }})](https://www.yoctoproject.org/)

The [Yocto Project](https://www.yoctoproject.org/) is pleased to report that it's core metadata (OpenEmbedded-Core) is now reproducible for all recipes (100% coverage) after issues with newer languages such as Golang were resolved. This was announced in their recent [Year in Review](https://www.yoctoproject.org/2021-a-year-in-review/) publication. It is of particular interest for security updates so that systems can have specific components updated but reducing the risk of other unintended changes and making the sections of the system changing very clear for audit.

The project is now also making heavy use of "equivalence" of build output to determine whether further items in builds need to be rebuilt or whether cached previously built items can be used. As mentioned in the article above, there are now public servers sharing this equivalence information. Reproducibility is key in making this possible and effective to reduce build times/costs/resource usage. 

<br>

## [*diffoscope*](https://diffoscope.org)

[![]({{ "/images/reports/2022-02/diffoscope.svg#right" | relative_url }})](https://diffoscope.org)

[*diffoscope*](https://diffoscope.org) is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions [`203`](https://diffoscope.org/news/diffoscope-203-released/), [`204`](https://diffoscope.org/news/diffoscope-204-released/), [`205`](https://diffoscope.org/news/diffoscope-205-released/) and [`206`](https://diffoscope.org/news/diffoscope-206-released/) to Debian *unstable*, as well as made the following changes to the code itself:

* Bug fixes:

    * Fix a `file(1)`-related regression where Debian `.changes` files that contained non-ASCII text were not identified as such, therefore resulting in seemingly arbitrary packages not actually comparing the nested files themselves. The non-ASCII parts were typically in the `Maintainer` or in the changelog text. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/428bbfaa)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/061698e6)]
    * Fix a regression when comparing directories against non-directories. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/94d08db4)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/ceb90b5d)]
    * If we fail to scan using `binwalk`, return `False` from `BinwalkFile.recognizes`. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/9e1786fa)]
    * If we fail to import `binwalk`, don't report that we are missing the Python `rpm` module! [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/dd7bace5)]

* Testsuite improvements:

    * Add a test for recent `file(1)` issue regarding `.changes` files. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/cb42460a)]
    * Use our `assert_diff` utility where we can within the `test_directory.py` set of tests. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/fc46f9f1)]
    * Don't run our `binwalk`-related tests as root or `fakeroot`. The latest version of `binwalk` has some new security protection against this. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2de5369c)]

* Codebase improvements:

    * Drop the `_PATH` suffix from module-level globals that are not paths. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/b5d46b42)]
    * Tidy some control flow in `Difference._reverse_self`. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/c2be3195)]
    * Don't print a warning to the console regarding `NT_GNU_BUILD_ID` changes. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/b0fb9c56)]

In addition, Mattia Rizzolo updated the Debian packaging to ensure that `diffoscope` and `diffoscope-minimal` packages have the same version.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f8b740ec)]

<br>

## Debian-related updates

[![]({{ "/images/reports/2022-02/debian.png#right" | relative_url }})](https://debian.org/)

Vagrant Cascadian wrote to the [`debian-devel`](https://lists.debian.org/debian-devel/) mailing list after noticing that the [`binutils` source package contained unreproducible logs in one of its binary packages](https://bugs.debian.org/950585). Vagrant expanded the discussion to one about *all* kinds of build metadata in packages and outlines a number of potential solutions that support reproducible builds and arbitrary metadata.

Vagrant also [started a discussion on `debian-devel`](https://lists.debian.org/debian-devel/2022/02/msg00050.html) after identifying a large number of [packages that embed build paths via RPATH when building with CMake](https://tests.reproducible-builds.org/debian/issues/unstable/cmake_rpath_contains_build_path_issue.html), including a list of packages (grouped by Debian maintainer) affected by this issue. Maintainers were requested to check whether their package still builds correctly when passing the `-DCMAKE_BUILD_RPATH_USE_ORIGIN=ON` directive.

On [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) this month, *kpcyrd* [announced the release of *rebuilderd-debian-buildinfo-crawler*](https://lists.reproducible-builds.org/pipermail/rb-general/2022-February/002477.html) a tool to parse the `Packages.xz` Debian package index file, attempts to discover the right `.buildinfo` file from [buildinfos.debian.net](https://buildinfos.debian.net ) and outputs it in a format that can be understood by [*rebuilderd*](https://github.com/kpcyrd/rebuilderd). The tool, [which is available on GitHub](https://github.com/kpcyrd/rebuilderd-debian-buildinfo-crawler), solves a [problem regarding correlating Debian version numbers with their builds](https://vulns.xyz/2022/01/debian-missing-version-string/).

*bauen1* provided two patches for [*debian-cd*](https://salsa.debian.org/images-team/debian-cd), the software used to make Debian installer images. This involved passing `--invariant` and `-i deb00001` to `mkfs.msdos(8)` and avoided embedding timestamps into the gzipped `Packages` and `Translations` files. After some discussion, the [patches in question](https://salsa.debian.org/images-team/debian-cd/-/merge_requests/22) were merged and will be included in *debian-cd* version 3.1.36.

Roland Clobus wrote another [in-depth status update](https://lists.reproducible-builds.org/pipermail/rb-general/2022-February/002482.html) about status of 'live' Debian images, summarising the current situation that "all major desktops build reproducibly with *bullseye*, *bookworm* and *sid*".

[![]({{ "/images/reports/2022-02/python-logo.png#right" | relative_url }})](https://bugs.debian.org/1004558)

The `python3.10` package was uploaded to Debian by *doko*, fixing an issue where [`.pyc` files were not reproducible because [the elements in `frozenset` data structures](https://bugs.debian.org/1004558) were not ordered reproducibly. This meant that to creating a bit-for-bit reproducible Debian chroot which included `.pyc` files was not reproducible. As of writing, the only remaining unreproducible parts of a `standard` chroot is `man-db`, but Guillem Jover has a patch for `update-alternatives` which will likely be part of the next release of `dpkg`.

Elsewhere in Debian, 139 reviews of Debian packages were added, 29 were updated and 17 were removed this month adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). A large number of issue types have been updated too, including the addition of [`captures_kernel_variant`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/185f62e5), [`erlang_escript_file`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/896eafd7), [`captures_build_path_in_r_rdb_rds_databases`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/6c83218d), [`captures_build_path_in_vo_files_generated_by_coq`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/778fab0e) and [`build_path_in_vo_files_generated_by_coq`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/82bea314).

<br>


## Website updates

[![]({{ "/images/reports/2022-02/website.png#right" | relative_url }})](https://reproducible-builds.org/)

There were quite a few changes to the [Reproducible Builds website and documentation](https://reproducible-builds.org/) this month as well, including:

* Chris Lamb:

    * Considerably rework the [*Who is involved?*]({{ "/contribute/" | relative_url }}) page. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0057f15f)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/34aff40b)]
    * Move the `contributors.sh` Bash/shell script into a Python script. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/71c16263)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/e28eef60)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2264d801)]

* Daniel Shahaf:

    * Try a different Markdown footnote content syntax to work around a rendering issue.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b3aae168)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/245f9bf7)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3d3d2bc6)]

* Holger Levsen:

    * Make a huge number of changes to the [*Who is involved?*]({{ "/contribute/" | relative_url }}) page, including pre-populating a large number of contributors who cannot be identified from the metadata of the website itself.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9373c264)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5fe5f72b)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/907fb5d9)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8d3f84e9)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5aa1f706)]
    * Improve linking to sponsors in sidebar navigation.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/96f12e4d)]
    * drop sponsors paragraph as the navigation is clearer now.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9bb6374b)]
    * Add [Mullvad VPN](https://mullvad.net) as a bronze-level sponsor .&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3f2750f4)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/bffea23b)]

* Vagrant Cascadian:

    * Remove a stray parenthesis from the [*Who is involved?*]({{ "/contribute/" | relative_url }}) page.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/00c4c2f3)]

<br>

## Upstream patches

The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. February's patches included the following:

* Bernhard M. Wiedemann:

    * [`btop`](https://github.com/aristocratos/btop/pull/266) (sort-related issue)
    * [`complexity`](https://build.opensuse.org/request/show/950695) (date)
    * [`giac`](https://build.opensuse.org/request/show/950705) (update the version with upstreamed date patch)
    * [`htcondor`](https://github.com/htcondor/htcondor/pull/455) (use [CMake](https://cmake.org/) timestamp)
    * [`libint`](https://build.opensuse.org/request/show/953236) (`readdir` system call related)
    * [`libnet`](https://build.opensuse.org/request/show/952874) (date-related issue)
    * [`librime-lua`](https://build.opensuse.org/request/show/953987) (sort filesystem ordering)
    * [`linux_logo`](https://build.opensuse.org/request/show/952927) (sort-related issue)
    * [`micro-editor`](https://build.opensuse.org/request/show/951851) (date-related issue)
    * [`openvas-smb`](https://build.opensuse.org/request/show/952933) (date-related issue)
    * [`ovmf`](https://build.opensuse.org/request/show/955619) (sort-related issue)
    * [`paperjam`](https://build.opensuse.org/request/show/952928) (date-related issue)
    * [`python-PyQRCode`](https://build.opensuse.org/request/show/950744) (date-related issue)
    * [`quimb`](https://github.com/jcmgray/quimb/issues/109) (single-CPU build failure)
    * [`radare2`](https://github.com/radareorg/radare2/pull/19699) ([Meson](https://mesonbuild.com/) date/time-related issue)
    * [`radare2`](https://github.com/radareorg/radare2/pull/19705) (Rework [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/) usage to be portable)
    * [`siproxd`](https://github.com/hb9xar/siproxd/pull/2) (date, with Sebastian Kemper + [follow-up](https://github.com/hb9xar/siproxd/pull/3)
    * [`xonsh`](https://build.opensuse.org/request/show/950673) ([Address Space Layout Randomisation](https://en.wikipedia.org/wiki/Address_space_layout_randomization)-related issue)
    * [`xsnow`](https://sourceforge.net/p/xsnow/tickets/10/) (date & `tar(1)`-related issue)
    * [`zip`](https://build.opensuse.org/request/show/954053) (toolchain issue related to filesystem ordering)

* Chris Lamb:

    * [#1005029](https://bugs.debian.org/1005029) filed against [`ltsp`](https://tracker.debian.org/pkg/ltsp) ([forwarded upstream](https://github.com/ltsp/ltsp/pull/660)).
    * [#1005197](https://bugs.debian.org/1005197) filed against [`pcmemtest`](https://tracker.debian.org/pkg/pcmemtest).
    * [#1005825](https://bugs.debian.org/1005825) filed against [`hatchling`](https://tracker.debian.org/pkg/hatchling).
    * [#1005826](https://bugs.debian.org/1005826) filed against [`mpl-sphinx-theme`](https://tracker.debian.org/pkg/mpl-sphinx-theme) ([forwarded upstream](https://github.com/matplotlib/mpl-sphinx-theme/pull/25))
    * [#1005827](https://bugs.debian.org/1005827) filed against [`gap-hapcryst`](https://tracker.debian.org/pkg/gap-hapcryst).
    * [#1005901](https://bugs.debian.org/1005901) filed against [`tree-puzzle`](https://tracker.debian.org/pkg/tree-puzzle).
    * [#1005954](https://bugs.debian.org/1005954) filed against [`jcabi-aspects`](https://tracker.debian.org/pkg/jcabi-aspects).
    * [#1005955](https://bugs.debian.org/1005955) filed against [`paper-icon-theme`](https://tracker.debian.org/pkg/paper-icon-theme).

* Roland Clobus:

    * [#1006358](https://bugs.debian.org/1006358) filed against [`libxmlb`](https://tracker.debian.org/pkg/libxmlb).

* Vagrant Cascadian:

    * [#1005408](https://bugs.debian.org/1005408) filed against [`wcwidth`](https://tracker.debian.org/pkg/wcwidth).
    * [#1005420](https://bugs.debian.org/1005420) filed against [`xir`](https://tracker.debian.org/pkg/xir).
    * [#1005421](https://bugs.debian.org/1005421) filed against [`xir`](https://tracker.debian.org/pkg/xir).
    * [#1005726](https://bugs.debian.org/1005726) filed against [`ruby-github-markup`](https://tracker.debian.org/pkg/ruby-github-markup).
    * [#1005727](https://bugs.debian.org/1005727) filed against [`ruby-tioga`](https://tracker.debian.org/pkg/ruby-tioga).
    * [#1005792](https://bugs.debian.org/1005792) filed against [`btop`](https://tracker.debian.org/pkg/btop).
    * [#1005793](https://bugs.debian.org/1005793) filed against [`libadwaita-1`](https://tracker.debian.org/pkg/libadwaita-1).
    * [#1005794](https://bugs.debian.org/1005794) filed against [`snibbetracker`](https://tracker.debian.org/pkg/snibbetracker).
    * [#1006252](https://bugs.debian.org/1006252) filed against [`cctbx`](https://tracker.debian.org/pkg/cctbx).
    * [#1006254](https://bugs.debian.org/1006254) filed against [`mdnsd`](https://tracker.debian.org/pkg/mdnsd).
    * [#1006256](https://bugs.debian.org/1006256) filed against [`gmerlin`](https://tracker.debian.org/pkg/gmerlin).
    * [#1006302](https://bugs.debian.org/1006302) filed against [`beav`](https://tracker.debian.org/pkg/beav).
    * [#1006385](https://bugs.debian.org/1006385) filed against [`krita`](https://tracker.debian.org/pkg/krita).
    * [#1006407](https://bugs.debian.org/1006407) filed against [`qt6-base`](https://tracker.debian.org/pkg/qt6-base).
    * [#1006455](https://bugs.debian.org/1006455) filed against [`onevpl-intel-gpu`](https://tracker.debian.org/pkg/onevpl-intel-gpu).
    * [#1006471](https://bugs.debian.org/1006471) filed against [`ruby3.0`](https://tracker.debian.org/pkg/ruby3.0).
    * [#1006473](https://bugs.debian.org/1006473) filed against [`nix`](https://tracker.debian.org/pkg/nix).
    * [#1006474](https://bugs.debian.org/1006474) filed against [`foma`](https://tracker.debian.org/pkg/foma).
    * [#1006476](https://bugs.debian.org/1006476) filed against [`ruby3.0`](https://tracker.debian.org/pkg/ruby3.0).

<br>

## Testing framework

[![]({{ "/images/reports/2022-02/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

The Reproducible Builds project runs a significant testing framework at [tests.reproducible-builds.org](https://tests.reproducible-builds.org), to check packages and other artifacts for reproducibility. This month, the following changes were made:

* Daniel Golle:

    * Update the OpenWrt configuration to not depend on the host LLVM, adding lines to the `.config` seed to build LLVM for eBPF from source.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/95f0a5db)]
    * Preserve more OpenWrt-related build artifacts.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/93d05a2e)]

* Holger Levsen:

* Temporary use a different Git tree when building OpenWrt as our tests had been broken since September 2020. This was reverted after the [patch in question](https://github.com/openwrt/openwrt/commit/0d25db7f17efbf5ab539508dd0a5d1eb739a1c43) was accepted by Paul Spooren into the canonical `openwrt.git` repository the next day. 
    * Various improvements to debugging OpenWrt reproducibility.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/786af187)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e07cd74a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2a2b9854)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/24642e74)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/017f92eb)]
    * Ignore `useradd` warnings when building packages.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/11321f75)]
    * Update the script to powercycle `armhf` architecture nodes to add a hint to where nodes named `virt-*`.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7d6af29b)]
    * Update the node health check to also fix failed `logrotate` and `man-db` services.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b6e16d39)]

* Mattia Rizzolo:

    * Update the website job after `contributors.sh` script was rewritten in Python.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f0300654)]
    * Make sure to set the `DIFFOSCOPE` environment variable when available.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c6de3ab7)]

* Vagrant Cascadian:

    * Various updates to the *diffoscope* timeouts.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8d480945)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a2baddb7)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b269be01)]

Node maintenance was also performed by Holger Levsen&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1a072358)] and Vagrant Cascadian&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/23a9c5d9)].

<br>

## Finally...

If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)