--- layout: report year: "2020" month: "03" title: "Reproducible Builds in March 2020" draft: false date: 2020-04-07 09:30:21 --- **Welcome to the March 2020 report from the [Reproducible Builds](https://reproducible-builds.org) project.** In our reports we outline the most important things that we have been up to over the past month and some plans for the future. ##### What are reproducible builds? [![]({{ "/images/reports/2020-03/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.
## News [![]({{ "/images/reports/2020-03/report.png#right" | relative_url }})](https://reproducible-builds.org/files/ReproducibleSummit5EventDocumentation.html) The report from our recent [summit in Marrakesh]({{ "/events/Marrakesh2019/" | relative_url }}) was published and is now available in both [PDF](https://reproducible-builds.org/files/ReproducibleSummit5EventDocumentation.pdf) and [HTML](https://reproducible-builds.org/files/ReproducibleSummit5EventDocumentation.html) formats. A sincere thank you to all of the Reproducible Builds community for the input to the event a sincere thank you to [Aspiration](https://aspirationtech.org/) for preparing and collating this report. [Harmut Schorrig](https://vishia.org/) published a detailed document on how to compile Java applications in such as way that the [`.jar` build artefact is reproducible across builds](https://vishia.org/Java/html5/source+build/reproducibleJar.html). A practical and hands-on guide, it details how to avoid unnecessary differences between builds by explicitly declaring an encoding as the default value differs across Linux and MS Windows systems and ensuring that the generated `.jar` — a variant of a `.zip` archive — does not embed any nondeterministic filesystem metadata, and so on. [Janneke](https://octodon.social/@janneke) gave a quick presentation on [GNU Mes](https://www.gnu.org/software/mes/) and reproducible builds during the lighting talk session at [LibrePlanet 2020](https://libreplanet.org/2020/). [[...](https://twitter.com/janneke_gnu/status/1239271789911592964)] [![]({{ "/images/reports/2020-03/scale-talk.jpeg#right" | relative_url }})](https://youtu.be/wRmOOKugpTc?t=19053) Vagrant Cascadian presented [*There and Back Again, Reproducibly!*](https://www.socallinuxexpo.org/scale/18x/presentations/there-and-back-again-reproducibly) [video](https://youtu.be/wRmOOKugpTc?t=19053) at [SCaLE 18x](https://www.socallinuxexpo.org/scale/18x) in Pasadena in California which generated [some attention on Twitter](https://twitter.com/pleia2/status/1236797044209008641). Hervé Boutemy [mentioned on our mailing list](https://lists.reproducible-builds.org/listinfo/rb-general/) in a thread titled [*Rebuilding and checking Reproducible Builds from Maven Central repository*](https://lists.reproducible-builds.org/pipermail/rb-general/2020-March/001862.html) that since the update of a central build script (the "parent [POM](https://maven.apache.org/guides/introduction/introduction-to-the-pom.html)") every Apache project using the Maven build system should build reproducibly. A [follow-up discussion](https://lists.apache.org/thread.html/ra05a971a2de961d27691bd4624850a06a862b4223116c0c904be8397%40%3Cdev.maven.apache.org%3E) regarding how to perform such rebuilds was also started on the Apache mailing list. [![]({{ "/images/reports/2020-03/telegram.png#right" | relative_url }})](https://telegram.org) The [Telegram](https://telegram.org/) instant-messaging platform [announced that they had updated their iOS and Android OS applications](https://twitter.com/TelegramBeta/status/1244639594810871809) and claim that they are reproducible according to [their full instructions](https://core.telegram.org/reproducible-builds), verifying that its original source code is exactly the same code that is used to build the versions available on the Apple App Store and Google Play distribution platforms respectfully. Hervé Boutemy [also reported](https://lists.reproducible-builds.org/pipermail/rb-general/2020-March/001869.html) about a new project called [`reproducible-central`](https://github.com/jvm-repo-rebuild/reproducible-central) which aims to allow anyone to rebuild a component from the [Maven Central Repository](https://search.maven.org/) that is expected to be reproducible and check that the result is as expected. [![]({{ "/images/reports/2020-03/dettrace.jpeg#right" | relative_url }})](https://youtu.be/YkmS-vf12nE) In [last month's report]({{ "/repots/2020-02/" }}) we detailed [Omar Navarro Leija](https://gatowololo.github.io/)'s work in and around an academic paper titled [*Reproducible Containers*](https://gatowololo.github.io/resources/publications/dettrace.pdf) which describes in detail the workings of a user-space container tool called [`dettrace`](https://github.com/dettrace/dettrace) ([PDF](https://gatowololo.github.io/resources/publications/dettrace.pdf)). Since then, the PhD student from the [University Of Pennsylvania](https://home.www.upenn.edu/) presented on this tool at the [ASPLOS 2020](https://asplos-conference.org/) conference in Lausanne, Switzerland. Furthermore, there were contributions to `dettrace` from the Reproducible Builds community itself. [[...](https://github.com/dettrace/dettrace/pull/278)][[...](https://github.com/dettrace/dettrace/pull/277)]
## Distribution work #### [openSUSE](https://www.opensuse.org/) [![]({{ "/images/reports/2020-03/opensuse.png#right" | relative_url }})](https://www.opensuse.org/) In [openSUSE](https://www.opensuse.org/), Bernhard M. Wiedemann published his [monthly Reproducible Builds status update](https://lists.opensuse.org/opensuse-factory/2020-04/msg00026.html) as well as made the following changes within the distribution itself: * [`avfs`](https://bugzilla.opensuse.org/show_bug.cgi?id=1168126) (report build problem in `%post` script) * [`arj`](https://build.opensuse.org/request/show/788351) (fix incorrect use of `strcpy`, [submitted upstream](https://sourceforge.net/p/arj/git/merge-requests/1/)) * [`brickv`](https://build.opensuse.org/request/show/788096) (update get [upstream fix](https://github.com/Tinkerforge/brickv/pull/23)) * [`fvwm-themes`](https://build.opensuse.org/request/show/789880) (delta between architectures in `noarch` package) * [`libpeas`](https://bugzilla.opensuse.org/show_bug.cgi?id=1165442) (report build failure in single-CPU mode) * [`pmix`](https://build.opensuse.org/request/show/788084) (update to incoporate [upstream fix](https://github.com/openpmix/openpmix/pull/1560)) * [`pw3270`](https://build.opensuse.org/request/show/788088) (date variation, [forwarded upstream](https://github.com/PerryWerneck/pw3270/pull/2)) * [`python-mailmanclient`](https://bugzilla.opensuse.org/show_bug.cgi?id=1165453) (report build failure in single-CPU mode) * [`ripgrep`](https://build.opensuse.org/request/show/788111) (CPU, [forwarded upstream](https://github.com/BurntSushi/ripgrep/commit/12e41809850a4ac14ed200101ef8b033d2a20c38)) * [`tensorflow2`](https://build.opensuse.org/request/show/787621) (avoid random temporary directory path) * [`tesseract-ocr`](https://build.opensuse.org/request/show/788680) (drop "native" architecture optimisations) * [`vlc`](https://build.opensuse.org/request/show/790372) (fixed "ghost" file size and sort archive, [already upstream](https://github.com/videolan/vlc/commit/87ea3c0dfb7367b434f688d657f931c074bb34f4)) #### [Debian](https://debian.org/) [![]({{ "/images/reports/2020-03/debian.png#right" | relative_url }})](https://debian.org/) Chris Lamb further refined his merge request for the `debian-installer` component to allow all arguments from `sources.list` files (such as "`[check-valid-until=no]`") in order that we can test the reproducibility of the installer images on the [Reproducible Builds own testing infrastructure](https://tests.reproducible-builds.org/debian/reproducible.html). ([#13](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13)) Holger Levsen filed a number of bug reports against the [`debrebuild`](https://salsa.debian.org/debian/devscripts/-/blob/master/scripts/debrebuild.pl) tool that attempts to rebuild a Debian package given a `.buildinfo` file as input, including: * Accepting signed [`.buildinfo`](https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles) files. ([#955050](https://bugs.debian.org/955050)) * Two [sbuild](https://wiki.debian.org/sbuild)-related bugs. ([#955123](https://bugs.debian.org/955123) & [#955304](https://bugs.debian.org/955304)) * Specific adjustments to the [APT](https://wiki.debian.org/Apt) configuration. ([#955307](https://bugs.debian.org/955307), [#955298](https://bugs.debian.org/955298) & [#955280](https://bugs.debian.org/955280)) * Requests to improve the documentation in various ways. ([#955049](https://bugs.debian.org/955049) & [#955308](https://bugs.debian.org/955308)) 48 reviews of Debian packages were added, 17 were updated and 34 were removed this month adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Many issue types were noticed, categorised and updated by Chris Lamb, including: * [`nondeterministic_gtk_icon_cache`](https://tests.reproducible-builds.org/debian/issues/unstable/nondeterministic_gtk_icon_cache_issue.html) [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/0b871fa1)] * [`nondeterministic_ordering_in_documentation_generated_by_doxygen`](https://tests.reproducible-builds.org/debian/issues/unstable/nondeterministic_ordering_in_documentation_generated_by_doxygen_issue.html) [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/d5dec485)] * [`nondeterministic_vo_files_generated_by_coq`](https://tests.reproducible-builds.org/debian/issues/unstable/nondeterministic_vo_files_generated_by_coq_issue.html) [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/fd565305)] utput * [`randomness_in_browserify_lite_output`](https://tests.reproducible-builds.org/debian/issues/unstable/randomness_in_browserify_lite_output_issue.html) [[...](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/633da098)] Finally, Holger opened a bug report against the software running [tracker.debian.org](https://tracker.debian.org/), a service for Debian Developers to follow the evolution of packages via web and email interfaces to request that they integrate information from [`buildinfos.debian.net`](https://buildinfos.debian.net) ([#955434](https://bugs.debian.org/955434)) and Chris Lamb kept [isdebianreproducibleyet.com](https://isdebianreproducibleyet.com) up to date. [[...](https://github.com/lamby/isdebianreproducibleyet.com/commits?author=lamby&since=2020-03-01T00:00:00Z&until=2020-04-01T00:00:00Z)]
## Software development #### [diffoscope](https://diffoscope.org) [![]({{ "/images/reports/2020-03/diffoscope.png#right" | relative_url }})](https://diffoscope.org) Chris Lamb made the following changes to [diffoscope](https://diffoscope.org), the Reproducible Builds project's in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading version `138` to Debian: * Improvements: * Don't allow errors with "[R](https://www.r-project.org/)" script deserialisation cause the entire operation to fail, for example if an external library cannot be loaded. [(#91](https://salsa.debian.org/reproducible-builds/diffoscope.git/issues/91)) * Experiment with memoising output from expensive external commands, eg. `readelf`. [(#93](https://salsa.debian.org/reproducible-builds/diffoscope.git/issues/93)) * Use `dumppdf` from the `python3-pdfminer` if we do not see any other differences from `pdftext`, etc. [(#92](https://salsa.debian.org/reproducible-builds/diffoscope.git/issues/92)) * Prevent a traceback when comparing two R `.rdx` files directly as the `get_member` method will return a file even if the file is missing. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/6344b5a)] * Reporting: * Display the supported file formats into the package long description. [(#90](https://salsa.debian.org/reproducible-builds/diffoscope.gi-/issues/90)) * Print a potentially-helpful message if the [PyPDF2](https://pythonhosted.org/PyPDF2/) module is not installed. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/e4712c8)] * Remove any duplicate comparator descriptions when formatting in the `--help` output or in the package long description. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/982d3a4)] * Weaken "*Install the X package to get a better output*" message to "*... may produce a better output*" as the former is not actually guaranteed. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/63983c2)] * Misc: * Ensure we only parse the recommended packages from `--list-debian-substvars` when we want them for `debian/tests/control` generation. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/b838101)] * Add [upstream metadata](https://wiki.debian.org/UpstreamMetadat) file [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/52e8838)] and add a Lintian override for [`upstream-metadata-in-native-source`](https://lintian.debian.org/tags/upstream-metadata-in-native-source.html) as "we" are upstream. [[...](https://salsa.debian.org/reproducible-builds/diffoscope.git/commit/253e662)] * Inline the `RequiredToolNotFound.get_package` method's functionality as it is only used once. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/889e10f)] * Drop the deprecated "`py36 = [..]`" argument in the `pyproject.toml` file. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/bf52b1b)] In addition, Vagrant Cascadian updated diffoscope in [GNU Guix](https://guix.gnu.org/) to version 138 [[...](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=03227eeb123bf038287ff07fd180004fd89b99fd)], as well as updating *reprotest* — our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences — to version 0.7.14 [[...](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=04fd952d54ffbc4935a44c50219be7c1da306531)]. #### Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month we wrote a large number of such patches, including: * Bernhard M. Wiedemann (via [openSUSE](https://www.opensuse.org/)): * [arj](https://sourceforge.net/p/arj/git/merge-requests/2/) (date variation) * [gnulib](https://lists.gnu.org/archive/html/bug-gnulib/2020-03/msg00024.html) (date variation) * [gnulib](https://lists.gnu.org/archive/html/bug-gnulib/2020-03/msg00061.html) (date variation) * [lasso](https://dev.entrouvert.org/issues/40454) (sort filesystem ordering) * [mono/at-spi-sharp](https://github.com/mono/mono/issues/19257) (report nondeterminism from filesystem nondeterminism) * [python-M2Crypto](https://gitlab.com/m2crypto/m2crypto/-/issues/275) (report security certs expiring in 2029) * [python-swifter](https://github.com/jmcarpenter2/swifter/issues/102) (report single-CPU build failure) * [QT uic](https://bugreports.qt.io/browse/QTBUG-83186) (report [ASLR](https://en.wikipedia.org/wiki/Address_space_layout_randomization) nondeterminism) * [tdiff](https://github.com/F-i-f/tdiff/issues/2) (report single-CPU build failure) * [tensorflow](https://github.com/tensorflow/tensorflow/issues/37997) (report ASLR-induced variation) * [volk](https://github.com/gnuradio/volk/pull/370) (drop compile-time CPU detection) * Chris Lamb: * [#952990](https://bugs.debian.org/952990) filed against [`pmemkv`](https://tracker.debian.org/pkg/pmemkv) ([forwarded upstream](https://github.com/pmem/pmemkv/pull/615)) * [#953071](https://bugs.debian.org/953071) filed against [`ndisc6`](https://tracker.debian.org/pkg/ndisc6). * [#953117](https://bugs.debian.org/953117) filed against [`infernal`](https://tracker.debian.org/pkg/infernal). * [#953263](https://bugs.debian.org/953263) filed against [`beep`](https://tracker.debian.org/pkg/beep). * [#953646](https://bugs.debian.org/953646) filed against [`node-nodedbi`](https://tracker.debian.org/pkg/node-nodedbi). * [#954409](https://bugs.debian.org/954409) filed against [`node-browserify-lite`](https://tracker.debian.org/pkg/node-browserify-lite). * [#955009](https://bugs.debian.org/955009) filed against [`font-manager`](https://tracker.debian.org/pkg/font-manager). * [#955287](https://bugs.debian.org/955287) filed against [`pdb2pqr`](https://tracker.debian.org/pkg/pdb2pqr). * [#955341](https://bugs.debian.org/955341) filed against [`gucharmap`](https://tracker.debian.org/pkg/gucharmap). * [#955364](https://bugs.debian.org/955364) filed against [`cloudkitty`](https://tracker.debian.org/pkg/cloudkitty). * [isbg](https://gitlab.com/isbg/isbg/-/issues/151) (report a non-deterministic documentation issue) #### Project documentation [![]({{ "/images/reports/2020-03/website.png#right" | relative_url }})]({{ "/" | relative_url }}) There was further work performed on [our documentation and website]({{ "/" | relative_url }}) this month including Alex Wilson adding [a section regarding using Gradle for reproducible builds]({{ "/docs/jvm/" | relative_url }}) in JVM projects [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5d0e646)] and Holger Levsen added the report from [our recent summit]({{ "/events/Marrakesh2019/" | relative_url }}) in Marrakesh [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/220770a)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f540070)]. In addition, Chris Lamb made a number of changes, including correcting the syntax of some CSS class formatting [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8b45a90)], improved some "filed against" copy a little better [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/27d677c)] and corrected a reference to [`calendar.monthrange`](https://docs.python.org/3/library/calendar.html#calendar.monthrange) Python method in a utility function. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website.git/commit/423203a)] #### Testing framework [![]({{ "/images/reports/2020-03/testframework.png#right" | relative_url }})](https://tests.reproducible-builds.org/) We operate a large and many-featured [Jenkins](https://jenkins.io/)-based testing framework that powers [`tests.reproducible-builds.org`](https://tests.reproducible-builds.org) that, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced. This month, Chris Lamb reworked the web-based package rescheduling tool to: * Require a HTTP `POST` method in the web-based scheduler as not only should [HTTP GET requests be idempotent](https://restfulapi.net/idempotent-rest-apis/) but this will allow many future improvements in the user interface. [[...](https://salsa.debian.org/reproducible-builds/jenkins.debian.net/commit/4e1682b2)][[...](https://salsa.debian.org/reproducible-builds/jenkins.debian.net/commit/f3b659d1)][[...](https://salsa.debian.org/reproducible-builds/jenkins.debian.net/commit/24822b26)] * Improve the authentication error message in said rescheduler to suggest that the developer's SSL certificate may have expired. [[...](https://salsa.debian.org/reproducible-builds/jenkins.debian.net/commit/e95f6baf)] In addition, Holger Levsen made the following changes: * Add a new [`ath97` subtarget](https://tests.reproducible-builds.org/openwrt/openwrt_ath97.html) for the [OpenWrt](https://openwrt.org/) distribution. * Revisit ordering of [Debian](https://debian.org/) suites; sort the *experimental* distribution last and reverse the ordering of suites to prioritise the suites in development. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0b84c43e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/04f40919)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4608f60e)] * Schedule Debian *buster* and *bullseye* a little less in order to allow *unstable* to catch up on the `i386` architecture. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/ce480e64)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5bed890d)] * Various cosmetic changes to the web-based scheduler. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fa5ba02e)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/bdbe00ce)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/cd0db406)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/494797ef)] * Improve wordings in the node health maintenance output. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/144cd64b)] Lastly, Vagrant Cascadian updated a link to the (formerly) weekly news to our reports page [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/cc3c9b04)] and *kpcyrd* fixed the escaping in an Alpine Linux inline patch [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e3624cd3)]. The usual build nodes maintenance was performed by Holger Levsen [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9a008f56)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fefd4228)], Mattia Rizzolo [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2f2b54ae)] and Vagrant Cascadian [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/54d3ab51)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/9b8dbef1)].
If you are interested in contributing to the Reproducible Builds project, please visit our [*Contribute*]({{ "/contribute/" | relative_url }}) page on our website. However, you can get in touch with us via: [![]({{ "/images/reports/2020-03/reproducible-builds.png#right" | relative_url }})](https://reproducible-builds.org/) * IRC: `#reproducible-builds` on `irc.oftc.net`. * Twitter: [@ReproBuilds](https://twitter.com/ReproBuilds) * Reddit: [/r/ReproducibleBuilds](https://reddit.com/r/reproduciblebuilds) * Mailing list: [`rb-general@lists.reproducible-builds.org`](https://lists.reproducible-builds.org/listinfo/rb-general)
This month's report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list. {: .small}