---
layout: report
year: "2019"
month: "07"
title: "Reproducible Builds in July 2019"
draft: false
date: 2019-08-05 16:06:30
---

**Welcome to the July 2019 report from the [Reproducible Builds](https://reproducible-builds.org) project!**
{: .lead}

[![]({{ "/images/reports/2019-07/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/)

In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.

The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

In July's report, we cover:

* **Front page** — *Media coverage, upstream news, etc.*
* **Distribution work** — *Shenanigans at DebConf19*
* **Software development** — *Software transparency, yet more diffoscope work, etc.*
* **On our mailing list** — *GNU tools, education and buildinfo files*
* **Getting in touch** — *... and how to contribute*

If you are interested in contributing to our project, we enthusiastically invite you to visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website.

---

## Front page

[![]({{ "/images/reports/2019-07/fdroid.png#right" | relative_url }})](https://f-droid.org/)

Nico Alt wrote a detailed and well-researched article titled "[*Trust is good, control is better*](https://nico.dorfbrunnen.eu/posts/2019/reproducibility-fdroid/)" which discusses Reproducible builds in [F-Droid](https://f-droid.org/) the alternative application repository for Android mobile phones. In contrast to the bigger commercial app stores F-Droid only offers apps that are free and open source software. The post not only demonstrates using [diffoscope](https://diffoscope.org) but talks more generally about how reproducible builds can prevent single developers or other important centralised infrastructure becoming targets for toolchain-based attacks.

Later in the month, F-Droid's aforementioned reproducibility status was mentioned on [episode 68](https://latenightlinux.com/late-night-linux-episode-68/) of the [Late Night Linux podcast](https://latenightlinux.com/). ([direct link](https://pca.st/D630#t=849) to 14:12)

[![]({{ "/images/reports/2019-07/thesis.png#right" | relative_url }})](http://bora.uib.no/handle/1956/20411)

Morten ("*Foxboron*") Linderud published his academic thesis "[*Reproducible Builds: break a log, good things come in trees*](http://bora.uib.no/handle/1956/20411)" which investigates and describes how transparency log overlays can provide additional security guarantees for computers automatically producing software packages. The thesis was part of Morten's studies at the [University of Bergen](https://www.uib.no/), Norway and is an extension of the work [New York University Tandon School of Engineering](https://engineering.nyu.edu/) has been doing with [package rebuilder integration in APT](https://www.youtube.com/watch?v=hbHa4OFv7Qo).

[Mike Hommey](https://glandium.org) posted to his blog about [*Reproducing the Linux builds of Firefox 68*](https://glandium.org/blog/?p=3923) which leverages that builds shipped by Mozilla should be reproducible from this version. He discusses the problems caused by the builds being optimised with [Profile-Guided Optimisation](https://en.wikipedia.org/wiki/Profile-guided_optimization) (PGO) but armed with the now-published profiling data, Mike provides [Docker](https://www.docker.com/)-based instructions how to reproduce the published builds yourself.

[![]({{ "/images/reports/2019-07/rust.jpg#left" | relative_url }})](https://www.rust-lang.org/)

Joel Galenson has been making progress on [a reproducible Rust compiler](https://github.com/jgalenson/reproducible-rustc) which includes support for a `--remap-path-prefix` argument, related to the concepts and problems involved in the [`BUILD_PATH_PREFIX_MAP`](https://reproducible-builds.org/specs/build-path-prefix-map/) proposal to fix issues with build paths being embedded in binaries.

Lastly, [Alessio Treglia](http://en.alessiotreglia.com/) posted to their blog about [*Cosmos Hub and Reproducible Builds*](http://en.alessiotreglia.com/articles/cosmos-hub-and-reproducible-builds/) which describes the reproducibility work happening in the [Cosmos Hub](https://hub.cosmos.network), a network of interconnected blockchains. Specifically, Alessio talks about work being done on the [Gaia](https://hub.cosmos.network/docs/what-is-gaia.html) development kit for the Hub.

<br>

---

### Distribution work

[![]({{ "/images/reports/2019-07/opensuse.png#right" | relative_url }})](https://www.opensuse.org/)

Bernhard M. Wiedemann posted his [monthly Reproducible Builds status update](https://lists.opensuse.org/opensuse-factory/2019-07/msg00364.html) for the [openSUSE](https://opensuse.org/) distribution where enabling. Enabling [Link Time Optimization](https://gcc.gnu.org/wiki/LinkTimeOptimization) (LTO) in this distribution's "[Tumbleweed](https://software.opensuse.org/distributions/tumbleweed)" branch caused multiple issues due to the number of cores on the build host being added to the `CFLAGS` variable. This affected, for example, [a `debuginfo/rpm` header](https://bugzilla.opensuse.org/show_bug.cgi?id=1140896) as well as resulted in [in `CFLAGS` appearing in built binaries](https://bugzilla.opensuse.org/show_bug.cgi?id=1141323) such as `fldigi`, `gmp`, `haproxy`, etc.

[![]({{ "/images/reports/2019-07/openwrt.png#right" | relative_url }})](https://openwrt.org/)

As highlighted in [last month's report]({{ "/reports/2019-07/" | relative_url }}), the [OpenWrt](https://openwrt.org/) project (a Linux operating system targeting embedded devices such as wireless network routers) [hosted a summit](http://lists.infradead.org/pipermail/openwrt-adm/2019-March/001021.html) in Hamburg, Germany. Their [full summit report and roundup](https://openwrt.org/meetings/hamburg2019/start) is now available that covers many general aspects within that distribution, including the work on reproducible builds that was done during the event.

#### Debian

[![]({{ "/images/reports/2019-07/debconf19.png#right" | relative_url }})](https://debconf19.debconf.org)

It was an extremely productive time in Debian this month in and around [DebConf19](https://debconf19.debconf.org/), the 20th annual conference for both contributors and users and was held at the Federal University of Technology in Paraná (UTFPR) in Curitiba, Brazil, from July 21 to 28. The conference was preceded by "DebCamp" from the 14th until the 19th with an additional "Open Day" that is targeted at the more-general public on the 20th.

There were a number of talks touching on the topic of reproducible builds and secure toolchains throughout the conference, including:

* *[Reproducible Builds - aiming for bullseye](https://debconf19.debconf.org/talks/30-reproducible-builds-aiming-for-bullseye/)* by Holger Levsen, Chris Lamb and Vagrant Cascadian.
* *[Software transparency: improving package manager security](https://debconf19.debconf.org/talks/66-software-transparency-improving-package-manager-security/)* presented by Benjamin Hof.
* *[Software transparency BoF](https://debconf19.debconf.org/talks/116-software-transparency-bof/)*, an informal "birds of a feather" session to discuss and collect ideas around detecting compromised archives.

There were naturally countless discussions regarding Reproducible Builds in and around the conference on the questions of tooling, infrastructure and our next steps as a project.

[![]({{ "/images/reports/2019-07/debian.png#center" | relative_url }})](https://debian.org/)

The release of Debian 10 *buster* has also meant the release cycle for the next release (codenamed "*bullseye*") has just begun. As part of this, the [Release Team recently announced](https://lists.debian.org/debian-devel-announce/2019/07/msg00002.html) that Debian will no longer allow binaries built and uploaded by maintainers on their own machines to be part of the upcoming release. This is great news not only for toolchain security in general but also in that it will ensure that all binaries that will form part of this release will likely have a `.buildinfo` file and thus metadata that could be used by others to reproduce and verify the builds.

Holger Levsen [filed a bug against the underlying tool](https://bugs.debian.org/932849) that maintains the Debian archive ("[dak](https://wiki.debian.org/DebianDak)") after he noticed that `.buildinfo` metadata files were not being automatically propagated if packages had to be manually approved or processed in the so-called "[`NEW` queue](https://ftp-master.debian.org/new.html)". After it was pointed out that the files were being retained in a separate location, Benjamin Hof [proposed a potential patch](https://bugs.debian.org/932849#22) for the issue which is pending review.

David Bremner posted to his blog post about "[Yet another buildinfo database](https://www.cs.unb.ca/~bremner//blog/posts/builtin-pho/)" that provides a SQL interface for querying `.buildinfo` attestation documents, particularly focusing on identifying packages that were built with a specific — and possibly buggy — build-dependency. Later at DebConf, David [demonstrated his tool live](https://meetings-archive.debian.net/pub/debian-meetings/2019/DebConf19/live-demos.webm) (starting at 36:30).

Ivo de Decker ("*ivodd"*) scheduled rebuilds of over 600 packages that last experienced an upload to the archive in December 2016 or earlier. This was so that they would be built using a version of the low-level `dpkg` package build tool that supports the generation of reproducible binary packages. The effect of this on the main archive will be deliberately staggered and thus visible throughout the upcoming weeks, potentially resulting in some of these packages now failing to build.

[Joaquin de Andres posted an update](https://lists.debian.org/debian-devel/2019/07/msg00613.html) regarding the work being done on continuous integration on [Debian's GitLab instance](https://salsa.debian.org) at DebConf19 in which he mentions, *inter alia*, a tool called [`atomic-reprotest`](https://salsa.debian.org/salsa-ci-team/atomic-reprotest/). This is a relatively new utility to help debug failures logged by our `reprotest` tool which attempts to test whether a build is reproducible or not. This tool was also mentioned [in a subsequent lightning talk](https://debconf19.debconf.org/talks/131-lightning-talks-2/).

Chris Lamb filed two bugs to drop the test jobs for both [`strip-nondeterminism`](https://tracker.debian.org/pkg/strip-nondeterminism) ([#932366](https://bugs.debian.org/932366)) and [`reprotest`](https://tracker.debian.org/pkg/reprotest) ([#932374](https://bugs.debian.org/932374)) after modifying them to build on the [Salsa](https://salsa.debian.org) server's own continuous integration platform and Holger Levsen shortly resolved them.

Lastly, 63 reviews of Debian packages were added, 72 were updated and 22 were removed this month, adding to [our large knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb added and categorised four new issue types, [`umask_in_java_jar_file`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/4e21b9d1), [`built_by-in_java_manifest_mf`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/e28c0c39), [`timestamps_in_manpages_generated_by_lopsubgen`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/9729b7fe) and [`codadef_coda_data_files`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/db675c20).

---

## Software development

The goal of [Benjamin Hof](https://www.net.in.tum.de/members/hof/)'s Software Transparency effort is to improve on the cryptographic signatures of the [APT package manager](https://en.wikipedia.org/wiki/APT_(Package_Manager)) by introducing a [Merkle tree-based](https://en.wikipedia.org/wiki/Merkle_tree) transparency log for package metadata and source code, in a similar vein to [certificate transparency](https://securitytrails.com/blog/what-are-certificate-transparency-logs). This month, he pushed [a number of repositories to our revision control system](https://salsa.debian.org/reproducible-builds/transparency) for further future development and review.

In addition, Bernhard M. Wiedemann updated his (deliberately) [unreproducible demonstration project](https://github.com/bmwiedemann/theunreproduciblepackage) to add [support for floating point variations](https://github.com/bmwiedemann/theunreproduciblepackage/commit/e5d59a3dda050b5c52b59af0ab610936d037c3b2) as well as [changes in the project's copyright year](https://github.com/bmwiedemann/theunreproduciblepackage/commit/c53ba1521867ca2b92ff93a367c5bfa90d7898f7).

#### Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

* Bernhard M. Wiedemann:
    * [`HSAIL-Tools`](https://build.opensuse.org/request/show/714387) — Sort hash, [submitted upstream](https://github.com/HSAFoundation/HSAIL-Tools/pull/54).
    * [`MozillaFirefox`](https://bugzilla.mozilla.org/show_bug.cgi?id=1568145) — Date.
    * [`MozillaFirefox`](https://build.opensuse.org/request/show/714438) ,[MozillaThunderbird](https://build.opensuse.org/request/show/714441) — Fix [a race condition](https://bugzilla.opensuse.org/show_bug.cgi?id=1137970).
    * [`blockattack`](https://github.com/blockattack/blockattack-game/pull/18) — Sort `zip` shell call and `gzip(1)`'s `-n`.
    * [`boswars`](https://build.opensuse.org/request/show/714579) — Sort Python `readdir`, [submitted upstream](https://savannah.nongnu.org/patch/index.php?9830).
    * [`bubbros`](https://build.opensuse.org/request/show/714457) — Strip `.png` date & time, [filed upstream](https://sourceforge.net/p/bub-n-bros/patches/3/).
    * [`calibre`](https://github.com/kovidgoyal/calibre/pull/1014) — Sort Python `glob`.
    * [`colobot-data`](https://github.com/colobot/colobot-data/pull/41) — Sort Python `readdir`.
    * [`cri-o`](https://github.com/cri-o/cri-o/pull/2643) — Date.
    * [`duckmarines`](https://build.opensuse.org/request/show/714601) — `zip -X`.
    * [`filtlan`](https://build.opensuse.org/request/show/717860) — Fix LaTeX run, dropping an unreproducible log with date.
    * [`frogatto`](https://build.opensuse.org/request/show/714463) — Strip `.png` date & time.
    * [`gcc`](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91307) — Report [Link-time optimisation](https://en.wikipedia.org/wiki/Interprocedural_optimization)-induced nondeterminism caused by using global constructors.
    * [`griefly`](https://github.com/griefly/griefly/pull/508) — Sort Python `readdir`.
    * [`herbstluftwm`](https://github.com/herbstluftwm/herbstluftwm/pull/563) — Use [CMake](https://cmake.org/) `TIMESTAMP` variable.
    * [`kitty`](https://github.com/kovidgoyal/kitty/pull/1804) — Sort filesystem.
    * [`lbreakout2`](https://build.opensuse.org/request/show/714451) — Strip `.png` date & time.
    * [`lsp-plugins`](https://github.com/sadko4u/lsp-plugins/pull/53) — Use `scandir` instead of `readdir` system call.
    * [`maven-javadoc-plugin`](https://issues.apache.org/jira/browse/MJAVADOC-619) — Report copyright using the current year.
    * [`metamail/mimegrep`](https://github.com/bitstreamout/mimegrep/pull/1) — Date.
    * [`mono`](https://bugzilla.opensuse.org/show_bug.cgi?id=1141502) — Report unreproducible [`.dll`](https://en.wikipedia.org/wiki/Dynamic-link_library) version.
    * [`netpanzer`](https://build.opensuse.org/request/show/714585) — Sort [SCons](https://scons.org/) `glob`/`readdir`.
    * [`notpacman`](https://build.opensuse.org/request/show/714453) — Strip `.png` date & time.
    * [`obs-build`](https://github.com/openSUSE/obs-build/pull/510) — Use `gzip -n` in Debian package build.
    * [`open-iscsi`](https://github.com/hreinecke/open-iscsi/pull/8) — Fix nondeterministic `%ghost` file size.
    * [`perl-File-Unpack`](https://build.opensuse.org/request/show/713417) — Fix a parallelism-induced race condition, [submitted upstream](https://github.com/jnweiger/perl-File-Unpack/pull/9).
    * [`python-futurist`](https://review.opendev.org/669130) — Drop Python `environment.pickle`.
    * [`python-geolib`](https://build.opensuse.org/request/show/713240) — Drop `environment.pickle`.
    * [`python-nautilus`](https://build.opensuse.org/request/show/714880) — Python date, [already filed upstream](https://github.com/GNOME/nautilus-python/pull/6).
    * [`python-pyreadstat`](https://github.com/Roche/pyreadstat/pull/37) — Sort Python readdir.
    * [`python-scikit-image`](https://build.opensuse.org/request/show/719287) — Drop randomness from package.
    * [`python-slycot`](https://build.opensuse.org/request/show/713579) — Drop unreproducible `.pyc` files.
    * [`python-statsmodels`](https://build.opensuse.org/request/show/719554) — Drop unreproducible `.pyc` files.
    * [`sienna`](https://build.opensuse.org/request/show/714584) — `zip -X`.
    * [`springrts`](https://build.opensuse.org/request/show/715002) — Use `strip-nondeterminism` on `.zip` modification times.
    * [`trilinos`](https://github.com/trilinos/Trilinos/pull/5559) — Sort a Perl `readdir` call / `File::Find`.
    * [`vdrift`](https://github.com/VDrift/vdrift/pull/168) — Fix a date exposed by in SCons and Python.
    * [`wordwarvi`](https://build.opensuse.org/request/show/714611) — Adjust a date, [already filed upstream](https://github.com/smcameron/wordwarvi/commit/c890eb38211741261f0e18692131ebfcddc847e8).
    * [`worldofpadman`](https://build.opensuse.org/request/show/714623) — Fix a date, [merged upstream](https://github.com/ioquake/ioq3/pull/414).
    * [`yadex`](https://build.opensuse.org/request/show/714615) — Fix file modification times.
    * [`znc`](https://build.opensuse.org/request/show/714939) — Avoid parallelism race from `CMakeFile`s.

* Chris Lamb:
    * [#931706](https://bugs.debian.org/931706) filed against [`node-d3-selection`](https://tracker.debian.org/pkg/node-d3-selection).
    * [#931854](https://bugs.debian.org/931854) filed against [`liblopsub`](https://tracker.debian.org/pkg/liblopsub).
    * [#932116](https://bugs.debian.org/932116) filed against [`snakemake`](https://tracker.debian.org/pkg/snakemake).
    * [#932117](https://bugs.debian.org/932117) filed against [`ninja-build`](https://tracker.debian.org/pkg/ninja-build).
    * [#932300](https://bugs.debian.org/932300) filed against [`sysvinit`](https://tracker.debian.org/pkg/sysvinit).
    * [#932301](https://bugs.debian.org/932301) filed against [`python-os-faults`](https://tracker.debian.org/pkg/python-os-faults).
    * [#932302](https://bugs.debian.org/932302) filed against [`calendar`](https://tracker.debian.org/pkg/calendar).
    * [#932365](https://bugs.debian.org/932365) filed against [`python-manilaclient`](https://tracker.debian.org/pkg/python-manilaclient).

[![]({{ "/images/reports/2019-07/fedora.png#right" | relative_url }})](https://getfedora.org/)

Neal Gompa, Michael Schröder & Miro Hrončok responded to [Fedora](https://getfedora.org)'s recent [change to `rpm-config`](https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/57) with some new developments within [rpm](https://github.com/rpm-software-management/rpm/pull/785) to fix an unreproducible "`Build Date`" and [reverted a change to the Python interpreter](https://github.com/fedora-python/cpython/pull/3) to switch back to unreproducible/time-based compile caches.

[![]({{ "/images/reports/2019-07/alpinelinux.png#right" | relative_url }})](https://www.alpinelinux.org)

Lastly, *kpcyrd* [submitted a pull request](https://github.com/alpinelinux/abuild/pull/93) for [Alpine Linux](https://alpinelinux.org/) to add [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/docs/source-date-epoch/) support to the [`abuild`](https://github.com/alpinelinux/abuild/) build tool in this operating system.

<br>

#### diffoscope

[![]({{ "/images/reports/2019-07/diffoscope.svg#right" | relative_url }})](https://diffoscope.org)

[diffoscope](https://diffoscope.org) is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. It is run countless times a day on [our testing infrastructure](https://tests.reproducible-builds.org/debian/reproducible.html) and is essential for identifying fixes and causes of non-deterministic behaviour.

This month, Chris Lamb made the following changes:

* Add support for Java `.jmod` modules ([#60](https://salsa.debian.org/reproducible-builds/diffoscope/issues/60)). However, not all versions of [`file(1)`](http://darwinsys.com/file/) support detection of these files yet, so we perform a manual comparison instead [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6a1769c)].
* If a command fails to execute but does not print anything to standard error, try and include the first line of [standard output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)) in the message we include in the difference. This was motivated by [`readelf(1)`](https://sourceware.org/binutils/docs/binutils/readelf.html) returning its error messages on standard output. [#59) [[...](https://salsa.debian.org/reproducible-builds/diffoscope/issues/59)]
* Add general support for `file(1)` 5.37 ([#57](https://bugs.debian.org/57)) but also adjust the code to not fail in tests when, eg, we do not have sufficiently newer or older version of `file(1)` ([#931881](https://bugs.debian.org/931881)).
* Factor out the ability to ignore the exit codes of `zipinfo` and `zipinfo -v` in the presence of non-standard headers. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/a98f743)] but only override the exit code from our special-cased calls to `zipinfo(1)` if they are `1` or `2` to avoid potentially masking real errors [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f1b6924)].
* Cease ignoring test failures in `stable-backports`. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/9c8cf9f)]
* Add missing textual `DESCRIPTION` headers for `.zip` and "Mozilla"-optimised `.zip` files. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/595e643)]
* Merge two overlapping environment variables into a single `DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS`. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/d5b9daf)]
* Update some reporting:
    * Re-add "return code" noun to "`Command foo exited with X`" error messages. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/fc262f2)]
    * Use `repr(..)`-style output when printing `DIFFOSCOPE_TESTS_FAIL_ON_MISSING_TOOLS` in skipped test rationale text. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/f903ae2)]
    * Skip the extra newline in `Output:\nfoo`. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2bf5591)]
* Add some explicit return values to appease [Pylint](https://www.pylint.org/), etc. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6a9faf8)]
* Also include the `python3-tlsh` in the Debian test dependencies. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/b99b03d)]
* Released and uploaded releasing versions 116, 117, 118, 119 & 120. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/2cfb1d0)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/39dff56)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/178d61a)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/02cc0c1)][[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/3c2f3f6)]

In addition, Marc Herbert provided a patch to catch failures to disassemble [ELF](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format) binaries.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope/issues/20)]

<br>

#### Project website

[![]({{ "/images/reports/2019-07/website.png#right" | relative_url }})](https://reproducible-builds.org/)

There was a yet more effort put into our [our website](https://reproducible-builds.org/) this month, including:

* Bernhard M. Wiedemann:
    * Update multiple works to use standard (or at least consistent) terminology.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/e815b1d)]
    * Document an alternative Python snippet in the [`SOURCE_DATE_EPOCH` examples]({{ "/docs/source-date-epoch/" | relative_url }}) examples.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ff02986)]
* Chris Lamb:
    * Split out our non-fiscal sponsors with a description [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/3e0bf9e)] and make them non-display three-in-a-row [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/04506b3)].
    * Correct references to [1&1 IONOS](https://www.ionos.co.uk/) (*née* Profitbricks). [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7cf91a8)]
    * Reduce ambiguity in our environment names. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f10dec3)]
    * Recreate the badge image, saving the `.svg` alongside it. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5fe94cd)]
    * Update our fiscal sponsors. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2537ab7)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/c15a143)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5ff48ee)]
    * Tidy the weekly reports section on the news page [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b9c2d01)], fixup the typography on the documentation page [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1e57e24)] and make all headlines stand out a bit more [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2d6f43d)].
    * Drop some old CSS files and fonts. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0c1c60f)]
    * Tidy news page a bit. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/acf3475)]
    * Fixup a number of issues in the report template and previous reports. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/57123d6)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/dba0dee)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/864a200)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/cc5cc92)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1207ef8)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b283da4)]

Holger Levsen also added explanations on how to install [diffoscope](https://diffoscope.org/) on OpenBSD&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/c4a35f3)] and FreeBSD&nbsp;[[...](https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/0c0dc6c)] to its homepage and Arnout Engelen added a preliminary and work-in-progress idea for a badge or "shield" program for upstream projects.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/67a4bde)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/68e7b42)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/fe67d8f)].

A special thank you to Alexander Borkowski&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/48aaa4d)] Georg Faerber&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/91a4e41)], and John Scott&nbsp;[[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a6a1100)] for their individual fixes. *To err is human; to reproduce, divine.*

<br>

#### strip-nondeterminism

[strip-nondeterminism](https://tracker.debian.org/pkg/strip-nondeterminism) is our tool to remove specific non-deterministic results from a completed build. This month, Niko Tyni provided a patch to use the Perl `Sub::Override` library for some temporary workarounds for issues in `Archive::Zip` instead of `Monkey::Patch` which was due for deprecation.&nbsp;[[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/8)].

In addition, Chris Lamb made the following changes:

* Identify data files from the COmmon Data Access (CODA) framework as being `.zip` files. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/6a40d5d)]
* Support OpenJDK ".jmod" files. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/11)]
* Pass `--no-sandbox` if necessary to bypass seccomp-enabled version of [`file(1)`](http://darwinsys.com/file/) which was causing a huge number of regressions in [our testing framework](http://tests.reproducible-builds.org/).
* Don't just run the tests but build the Debian package instead using Salsa's centralised scripts so that we get code coverage, Lintian, autopkgtests, etc. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/27b730e)][[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/6947d6f)]
* Update tests:
    * Don't build release Git tags on `salsa.debian.org`. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/7e1187a)]
    * Merge the `debian` branch into the `master` branch to simplify testing and deployment [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/b1132f4)] and update `debian/gbp.conf` to match [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/b948d76)].
* Drop misleading and outdated `MANIFEST` and `MANIFEST.SKIP` files as they are not used by our release process. [[...](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/368f18b)]

<br>

#### Test framework

[![]({{ "/images/reports/2019-07/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/)

We operate a comprehensive [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org). The following changes were performed in the last month:

* Holger Levsen:
    * [Debian](https://www.debian.org/)-specific changes:
        * Make a large number of adjustments to support the new Debian *bullseye* distribution and the release of *buster*.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7b0ddd60)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d7b64704)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a6e730e5)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b4cf99e7)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b96128f2)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3e39e65e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f6687d8e)]
[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b2a24dbe)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/55d0c8c1)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6087599a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e71c4a41)]
        * Fix the colours for the five suites now being built.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9e30c02e)]
        * Make a number code improvements to the calculation of our "metapackage" sets including refactoring and changes of email address, etc.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9b1773e1)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4ffcc496)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9d5745fd)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/26b7c366)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/300aea7e)]
        * Add the "`http-proxy`" variable to the displayed node info.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/055afc06)]

    * [Alpine](https://alpinelinux.org/) changes:
        * Rebuild the webpages every two hours (instead of twice per hour).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/095234d5)]

    * Reproducible tooling:
        * Fix the detection of version number in [Arch Linux](https://www.archlinux.org/).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d4d8adf1)]
        * Drop reprotest and strip-nondeterminism jobs as we run that via [Salsa CI](https://wiki.debian.org/Salsa/Doc#Running_Continuous_Integration_.28CI.29_tests) now.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/125d4d39)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2c9984a4)]
        * Add a link to current SQL database schema.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/29a45858)]

* Mattia Rizzolo:
    * Make a number of adjustments to support the new Debian *bullseye* distribution.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b95fa98)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b540da3)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/06b6534d)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8fd697c7)]
    * Ensure that our `arm64` hosts always trust the Debian archive keyring.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6e0fc050)]
    * Enable the [backports](https://backports.debian.org/) repositories on the `arm64` build hosts.&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/153479a2)]

Holger Levsen&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fe6d8aa0)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b2f841f3)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a18fdf72)] and Mattia Rizzolo&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/02f1e807)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0783e4ca)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8497c4cc)] performed the usual node maintenance and lastly, Vagrant Cascadian added support to generate a `reproducible-tracker.json` metadata file for the next release of Debian (*bullseye*).&nbsp;[[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ebf99061)]

---

## On the mailing list

Chris Lamb cross-posted his reply to the "[Re: file(1) now with seccomp support enabled](https://lists.reproducible-builds.org/pipermail/rb-general/2019-July/001612.html) thread that was [originally started on the `debian-devel`](https://lists.debian.org/debian-devel/2019/07/msg00391.html) Debian list. In his post, he refers to a strip-nondeterminism not being able to accommodate the additional security hardening in [`file(1)`](http://darwinsys.com/file/) and the changes made to the tool in order to do fix this issue which was causing a huge number of regressions in [our testing framework](http://tests.reproducible-builds.org/).

Matt Bearup wrote about his experience when he [generated different checksums for the `libgcrypt20` package](https://lists.reproducible-builds.org/pipermail/rb-general/2019-July/001606.html) which resulted in some pointers etc. in that one should be using the equivalent `.buildinfo` post-build certificate when attempting to reproduce any particular build.

[![]({{ "/images/reports/2019-07/gnu.png#right" | relative_url }})](https://www.science-community.org/en/node/203684)

Vagrant Cascadian posted a [request for comments regarding a potential proposal](https://lists.reproducible-builds.org/pipermail/rb-general/2019-July/001622.html) to the [GNU Tools "Cauldron"](https://www.science-community.org/en/node/203684) gathering to be held in Montréal, Canada during September 2019 and Bernhard M. Wiedemann posed a query about [using consistent terms](https://lists.reproducible-builds.org/pipermail/rb-general/2019-July/001603.html) on our webpages and elsewhere.

Lastly, in a thread titled "[Reproducible Builds - aiming for bullseye: comments and a purpose](https://lists.reproducible-builds.org/pipermail/rb-general/2019-July/001610.html)" Jathan asked about whether we had considered offering "101"-like beginner sessions to fix packages that are not currently reproducible.

<br>

---

## Getting in touch

If you are interested in contributing the Reproducible Builds project, please visit our [*Contribute*](https://reproducible-builds.org/contribute/) page on our website. However, you can get in touch with us via:

 * IRC: `#reproducible-builds` on `irc.oftc.net`.

 * Twitter: [**@ReproBuilds**](https://twitter.com/ReproBuilds)

 * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)

[![]({{ "/images/reports/2019-07/reproducible-builds.png#center" | relative_url }})](https://reproducible-builds.org/)

*This month's report was written by Benjamin Hof, Bernhard M. Wiedemann, Chris Lamb, Holger Levsen and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.*
{: .text-muted}