---
layout: report
year: "2020"
month: "02"
title: "Reproducible Builds in February 2020"
draft: false
date: 2020-03-06 18:27:51
---

**Welcome to the February 2020 report from the [Reproducible Builds](https://reproducible-builds.org) project.**
{: .lead}

[![]({{ "/images/reports/2020-02/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)

One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.

The motivation behind the reproducible builds effort is to provide the ability to demonstrate these binaries originated from a particular, trusted, source release: if identical results are generated from a given source in all circumstances, reproducible builds provides the means for multiple third-parties to reach a consensus on whether a build was compromised via distributed checksum validation or some other scheme.

In this month's report, we cover:

* **Media coverage & upstream news** — *A new paper on reproducible containers, Ruby updates, etc.*
* **Distribution work** — *More work in Debian, openSUSE & friends.*
* **Software development** — *Updates and improvements to our tooling.*
* **Getting in touch** — *How to contribute, more venues for discussion.*

If you are interested in contributing to the project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.
{: .small}

<br>

## Media coverage & upstream news

[![]({{ "/images/reports/2020-02/dettrace.png#right" | relative_url }})](https://gatowololo.github.io/resources/publications/dettrace.pdf)

[Omar Navarro Leija](https://gatowololo.github.io/), a PhD student at the [University Of Pennsylvania](https://home.www.upenn.edu/), published a paper entitled [*Reproducible Containers*](https://gatowololo.github.io/resources/publications/dettrace.pdf) that describes in detail the workings of a new user-space container tool called *DetTrace*:

> All computation that occurs inside a DetTrace container is a pure function of the initial filesystem state of the container. Reproducible containers can be used for a variety of purposes, including replication for fault-tolerance, reproducible software builds and reproducible data analytics. **We use DetTrace to achieve, in an automatic fashion, reproducibility for 12,130 Debian package builds, containing over 800 million lines of code, as well as bioinformatics and machine learning workflows.**

There was also considerable [discussion on our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2020-February/001820.html) regarding this research and a presentation based on the paper will occur at the [ASPLOS 2020](https://asplos-conference.org/) conference between March 16th — 20th in Lausanne, Switzerland.

[![]({{ "/images/reports/2020-02/fosdem.jpeg#center" | relative_url }})](https://fosdem.org/2020/schedule/event/debate_license_compliance/)

The many virtues of Reproducible Builds were touted as benefits for software compliance in a talk at [FOSDEM 2020](https://fosdem.org/2020/), debating whether the [Careful Inventory of Licensing Bill of Materials Have Impact of FOSS License Compliance](https://fosdem.org/2020/schedule/event/debate_license_compliance/) which pitted [Jeff McAffer](https://twitter.com/jeffmcaffer) and [Carol Smith](https://www.fossygirl.com/) against [Bradley Kuhn](http://ebb.org/bkuhn/) and Max Sills. (~47 minutes in).

Nobuyoshi Nakada updated the canonical implementation of the [Ruby programming language](https://www.ruby-lang.org/) a change such that filesystem [globs](https://en.wikipedia.org/wiki/Glob_(programming)) (ie. calls to list the contents of filesystem directories) will henceforth be [sorted in ascending order](https://bugs.ruby-lang.org/issues/8709#note-19). Without this change, the underlying nondeterministic ordering of the filesystem is exposed to the language which often results in an unreproducible build.

Vagrant Cascadian reported on [our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) regarding a [quick reproducible test for the GNU Guix distribution](https://lists.reproducible-builds.org/pipermail/rb-general/2020-February/001810.html), which resulted in 81.9% of packages registering as reproducible in his installation:

    $ guix challenge --verbose --diff=diffoscope ...
    2,463 store items were analyzed:
      - 2,016 (81.9%) were identical
      - 37 (1.5%) differed
      - 410 (16.6%) were inconclusive

[Jeremiah Orians announced on our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2020-February/001798.html) the release of a number of tools related to cross-compilation such as [`M2-Planet`](https://github.com/oriansj/M2-Planet) and [`mescc-tools-seed`](https://github.com/oriansj/mescc-tools-seed). This project attemps a full bootstrap of a cross-platform compiler for the C programming language (written in C itself) from hex, the ultimate goal being able to demonstrate fully-bootstrapped compiler from hex to the [GCC GNU Compiler Collection](https://gcc.gnu.org/). This has many implications in and around [Ken Thompson](https://en.wikipedia.org/wiki/Ken_Thompson)'s [*Trusting Trust*](https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf) attack outlined in Thompson's 1983 [Turing Award Lecture](https://amturing.acm.org/lectures.cfm).

Twitter user [@TheYoctoJester](https://twitter.com/TheYoctoJester) posted an executive summary of reproducible builds in the [Yocto Project](https://www.yoctoproject.org/):

[![]({{ "/images/reports/2020-02/TheYoctoJester.png#center" | relative_url }})](https://twitter.com/TheYoctoJester/status/1226242232543518720)

Finally, Reddit user [`tofflos`](https://www.reddit.com/user/tofflos) posted to the [/r/Java subreddit](https://www.reddit.com/r/java) asking about how to achieve [reproducible builds with Maven](https://www.reddit.com/r/java/comments/faqadv/reproducible_builds_with_maven/) and [Chris Lamb](http://chris-lamb.co.uk/) noticed that the Linux kernel documentation about reproducible builds of it is available on the [kernel.org](https://www.kernel.org) homepages [in an attractive HTML format](https://www.kernel.org/doc/html/latest/kbuild/reproducible-builds.html).

<br>

## Distribution work

#### [Debian](https://debian.org/)

[![]({{ "/images/reports/2020-02/debian.png#right" | relative_url }})](https://debian.org/)

Chris Lamb created a merge request for the core `debian-installer` package to allow all arguments and options from `sources.list` files (such as "`[check-valid-until=no]`", etc.) in order that we can test the reproducibility of the installer images on the [Reproducible Builds own testing infrastructure](https://tests.reproducible-builds.org/debian/reproducible.html).&nbsp;([#13](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13))

Thorsten Glaser [followed-up to a bug filed against the `dpkg-source` component](https://bugs.debian.org/796257#29) that was originally filed in late 2015 that claims that the build tool does not respect permissions when unpacking tarballs if the [umask](https://en.wikipedia.org/wiki/Umask) is set to `0002`.

Matthew Garrett posted to the [`debian-devel` mailing list](https://lists.debian.org/debian-devel/) on the topic of "[Producing verifiable initramfs images](https://lists.debian.org/debian-devel/2020/02/msg00121.html)" as part of a wider conversation on being able to trust the entire software stack on our computers.

59 reviews of Debian packages were added, 30 were updated and 42 were removed this month adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Many issue types were noticed and categorised by Chris Lamb, including:

* [`openstack_pkg_tools_python_shebang_and_dependencies`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/bebb8e68)
* [`build_path_in_code_generated_by_dgbus_codegen`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/2757cc3a)
* [`random_argument_handling_in_javatools_jh_build`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/55dcb305)
* [`fortran_captures_build_path`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/feef3e6f)
* *etc.*

#### [openSUSE](https://www.opensuse.org/)

[![]({{ "/images/reports/2020-02/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)

In [openSUSE](https://www.opensuse.org/), Bernhard M. Wiedemann published his [monthly Reproducible Builds status update](https://lists.opensuse.org/opensuse-factory/2020-03/msg00014.html) as well as provided the following patches:

* [`python-rpm-macros`](https://github.com/openSUSE/python-rpm-macros/pull/42) (do not save time-based `.pyc` files for tests)
* [`solfege`](https://build.opensuse.org/request/show/778645) (filesystem ordering issue sent upstream via email; package is [orphaned upstream](https://savannah.gnu.org/bugs/index.php?53159))
* [`DVDStyler`](https://build.opensuse.org/request/show/775706) (zip timestamps, [submitted upstream](https://sourceforge.net/p/dvdstyler/DVDStyler/merge-requests/1/))
* [`python-pikepdf`](https://build.opensuse.org/request/show/778615) (recreate unreproducible `.pyc` files)

<br>

## Software development

#### [diffoscope](https://diffoscope.org)

[![]({{ "/images/reports/2020-01/diffoscope.png#right" | relative_url }})](https://diffoscope.org)

[*diffoscope*](https://diffoscope.org) is our in-depth and content-aware diff-like utility that can locate and diagnose reproducibility issues. It is run countless times a day on [our testing infrastructure](https://tests.reproducible-builds.org/debian/reproducible.html) and is essential for identifying fixes and causes of nondeterministic behaviour.

Chris Lamb made the following changes this month, including uploading version `137` to Debian:

* The `sng` image utility appears to return with an exit code of 1 if there are even minor errors in the file.&nbsp;([#950806](https://bugs.debian.org/950806))
* Also extract `classes2.dex`, `classes3.dex` from `.apk` files extracted by `apktool`.&nbsp;([#88](https://salsa.debian.org/reproducible-builds/diffoscope/issues/88))
* No need to use `str.format` if we are just returning the string.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/adbd7f7)]
* Add generalised support for "ignoring" returncodes&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/dd23327)] and move special-casing of returncodes in zip to use `Command.VALID_RETURNCODES`.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/ad9100f)]

#### Other tools

*disorderfs* is our [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues. This month, Vagrant Cascadian updated the `Vcs-Git` to specify the `debian` packaging branch.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/disorderfs/commit/bab6d34)]

*reprotest* is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, versions `0.7.13` and `0.7.14` were [uploaded to Debian unstable](https://tracker.debian.org/pkg/reprotest) by Holger Levsen after Vagrant Cascadian added support for [GNU Guix](https://guix.gnu.org/)&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reprotest/commit/31bd4fe)].

#### Project documentation & website

[![]({{ "/images/reports/2020-02/website.png#right" | relative_url }})](https://reproducible-builds.org/)

There was more work performed on [our documentation and website](https://reproducible-builds.org/) this month. Bernhard M. Wiedemann added a Java [Gradle Build Tool](https://gradle.org/) snippet to the [`SOURCE_DATE_EPOCH`]({{ "/docs/SOURCE_DATE_EPOCH/" | relative_url }}) documentation&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/93dd5d5)] and normalised various terms to "unreproducible"&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3998506)].

[Chris Lamb](http://chris-lamb.co.uk/) added a [Meson.build](https://mesonbuild.com/) example&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/433f91f)] and improved the documentation for the [CMake](https://cmake.org/)&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6c0c999)] to the [`SOURCE_DATE_EPOCH`]({{ "/docs/SOURCE_DATE_EPOCH/" | relative_url }}) documentation, replaced "anyone can" with "anyone may" as, well, not everyone has the resources, skills, time or funding to actually do what it refers to&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0ddf47d)] and improved the pre-processing for our report generation&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/bd6bbe1)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/964052a)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/40b354d)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6b727d0)] etc.

In addition, Holger Levsen updated our [news page]({{ "/docs/news/" | relative_url }}) to improve the list of reports&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/35f17c3)], added an explicit mention of the weekly news time span&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9922a08)] and reverted sorting of news entries to have latest on top&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2187cee)] and Mattia Rizzolo added [Codethink](https://www.codethink.co.uk/) as a non-fiscal sponsor&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/11406ea)] and lastly Tianon Gravi added a [Docker Images](https://debuerreotype.github.io/) link underneath the "Debian" project on our "[Projects]({{ "/who/" | relative_url }})" page&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/691f971)].

#### Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

* Bernhard M. Wiedemann (for the [openSUSE](https://www.opensuse.org/) distribution):

    * [`rdiff-backup`](https://github.com/rdiff-backup/rdiff-backup/pull/258) — fixed, build date in Python manpage
    * [`click-man`](https://github.com/click-contrib/click-man/pull/37) — fixed, toolchain, build date in Python manpage
    * [`rpm`](https://github.com/rpm-software-management/rpm/issues/1056) — report filesystem-order related nondeterminism
    * [`python-enaml`](https://github.com/nucleic/enaml/issues/397) — report nondeterministic `.enamlc` compiler output
    * [`k9s`](https://github.com/derailed/k9s/pull/572) — date
    * [`python-xcffib`](https://github.com/tych0/xcffib/pull/100) — fix a test with a race condition

* Chris Lamb:

    * [#943956](https://bugs.debian.org/943956) re-opened against [`snakemake`](https://tracker.debian.org/pkg/snakemake).
    * [#950630](https://bugs.debian.org/950630) filed against [`designate`](https://tracker.debian.org/pkg/designate).
    * [#950936](https://bugs.debian.org/950936) filed against [`pynwb`](https://tracker.debian.org/pkg/pynwb).
    * [#950942](https://bugs.debian.org/950942) filed against [`python-oslo.reports`](https://tracker.debian.org/pkg/python-oslo.reports).
    * [#951357](https://bugs.debian.org/951357) filed against [`mate-desktop`](https://tracker.debian.org/pkg/mate-desktop) ([forwarded upstream](https://github.com/mate-desktop/mate-desktop/pull/430)).
    * [#951573](https://bugs.debian.org/951573) filed against [`javatools`](https://tracker.debian.org/pkg/javatools).
    * [#952493](https://bugs.debian.org/952493) filed against [`xavs2`](https://tracker.debian.org/pkg/xavs2).
    * [#952694](https://bugs.debian.org/952694) filed against [`snapd-glib`](https://tracker.debian.org/pkg/snapd-glib) ([forwarded upstream](https://github.com/snapcore/snapd-glib/pull/82)).
    * [#952762](https://bugs.debian.org/952762) filed against [`openstack-pkg-tools`](https://tracker.debian.org/pkg/openstack-pkg-tools).
    * [ccextractor](https://github.com/CCExtractor/ccextractor/issues/1230) — issues with MacOSX BSD date not understanding [GNU date](https://www.gnu.org/software/coreutils/manual/html_node/date-invocation.html) options
    * [`python-django`](https://salsa.debian.org/python-team/modules/python-django/commit/e43666ba7bdb8cbc73c6c738a11de44fb66fa9c0) (Always build the documentation in English)
    * [`python_example`](https://github.com/pybind/python_example/pull/53) (sort `glob` / `readdir`)

Vagrant Cascadian submitted patches via the Debian bug tracking system targeting the packages the [Civil Infrastructure Platform](https://www.cip-project.org/) has identified via the "[CIP](https://tests.reproducible-builds.org/debian/bullseye/amd64/pkg_set_CIP.html)" and "[CIP build depends](https://tests.reproducible-builds.org/debian/bullseye/amd64/pkg_set_CIP_build-depends.html)" package sets:

* [#950410](https://bugs.debian.org/950410) filed against [`autogen`](https://tracker.debian.org/autogen).
* [#950417](https://bugs.debian.org/950417) & [950416](https://bugs.debian.org/950416) & [950415](https://bugs.debian.org/950415) filed against [`autoconf`](https://tracker.debian.org/autoconf).
* [#950419](https://bugs.debian.org/950419) filed against [`m4`](https://tracker.debian.org/m4).
* [#950444](https://bugs.debian.org/950444) filed against [`intltool`](https://tracker.debian.org/intltool).
* [#950585](https://bugs.debian.org/950585) filed against [`binutils-dev`](https://tracker.debian.org/binutils-dev).
* [#950603](https://bugs.debian.org/950603) filed against [`yodl-doc`](https://tracker.debian.org/yodl-doc).
* [#950606](https://bugs.debian.org/950606) filed against [`gdb-source`](https://tracker.debian.org/gdb-source).
* [#950704](https://bugs.debian.org/950704) filed against [`automake-1.15`](https://tracker.debian.org/automake-1.15).
* [#951031](https://bugs.debian.org/951031) filed against [`libtool`](https://tracker.debian.org/libtool).


#### Testing framework

[![]({{ "/images/reports/2020-02/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

We operate a fully-featured and comprehensive [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org). This month, the following changes were made by Holger Levsen:

* [Debian](https://debian.org/):

    * Configured an instance of David Bremner's [`builtin-pho`](https://salsa.debian.org/bremner/builtin-pho/) database of `.buildinfo` files. This has resulted in so that we now know that [Debian bullseye contains 4,557 source packages for the `amd64` architecture without corresponding `.buildinfo` files](https://tests.reproducible-builds.org/debian/bullseye/amd64/index_no_buildinfos.html) and [25,668 source packages with `.buildinfo` files](https://tests.reproducible-builds.org/debian/bullseye/amd64/index_buildinfos.html).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fd3d4620)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b78a784e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/26db0a3a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/81993e7a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c733ee1d)]
    * Forward mails addressed to `buildinfo@` to the `root` user.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b26a4a98)]
    * Temporarily revert using backports&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/427ca05c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/71c2bf83)] and use `devscripts` from buster-backports&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3d72168d)].
    * Update URL for [PureOS](https://www.pureos.net/) package set.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a13affc8)]
    * Deprioritise the scheduling of older old packages in *bullseye* and *unstable*.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/89a95738)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/89830eca)]
    * Treat the *bullseye* distribution like *unstable* when scheduling the `i386` architecture, to allow the latter to catch up a bit.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4608f60e)]

* Misc/other:

    * Disable the last active [Alpine](https://alpinelinux.org/) builder too for now.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ebda0872)]
    * Improve generated HTML output.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/561184b2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0c24ca70)]
    * Ignore [Fedora](https://getfedora.org/) jobs.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/56d1ed08)]
    * Include links to failed and "unstable" jobs in HTML output.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/302122e3)]
    * Start weighting job importance and five a lot more weight to nodes acting as a proxy for others.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/19c2338b)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/69d0e001)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/23437d83)]
    * Add new job to calculate the overall system health for [tests.reproducible-builds.org](https://tests.reproducible-builds.org/) for usage with [Jelle van der Waa](https://vdwaa.nl/)'s [Reproducible Builds status display](https://github.com/jelly/reproduciblebuilds-display).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/aaca6d46)]

[![]({{ "/images/reports/2020-02/reprobuilds-display.jpeg#center" | relative_url }})](https://github.com/jelly/reproduciblebuilds-display)

In addition, Mattia Rizzolo added an [Apache](https://httpd.apache.org/) web server redirect for [buildinfos.debian.net](https://buildinfos.debian.net/)&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c4f3448a)] and reverted the reshuffling of `arm64` architecture builders&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1e0109c3)]. The usual build node maintenance was performed by Holger Levsen, Mattia Rizzolo&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0f42cb5c)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/51f0aeca)] and Vagrant Cascadian.

<br>

## Getting in touch

If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds)

 * Reddit: [/r/ReproducibleBuilds](https://reddit.com/r/reproduciblebuilds)

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)

<br>

---

This month's report was written by Bernhard M. Wiedemann, Chris Lamb and Holger Levsen. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
{: .small}