1. About jenkins.debian.net

See "about jenkins.debian.net" for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The the preferred form of contributions is patches via pull requests.

2. Fix user submitted bugs

3. General ToDo

  • replace amd64 in scripts with $HOSTARCH

  • extend /etc/rc.local to do cleanup of lockfiles

  • explain in README how to write jobs, e.g. which paths are on tmpfs

    • EXECUTOR_NUMBER for X

  • run all bash scripts with set -u and set -o pipefail: http://redsymbol.net/articles/unofficial-bash-strict-mode/

  • teach bin/chroot-*.sh and bin/d-i_build.sh how to nicely deal with network problems… (as both reproducible_build.sh and schroot-create.sh do)

  • salsa stuff:

    • triggers for d-i repos are missing

3.1. nodes at OSUOSL

  • five new nodes

    • tune diskspace needs

    • mention o4+5 in THANKS and explain usage

    • check kernels on all 5 nodes

    • mv snapshot.r-b.o from osuosl4 to osuosl5

      • setup xfs on o4

    • setup netconsoles: 

<guerby> | Ramereth, h01ger netconsole is about the only way to diagnose this kind of issue in my experience
<guerby> | h01ger, setup rsyslogd on one of the gccserver then it's just one modprobe netconsole
<guerby> | h01ger, since the machine are on the same LAN
<guerby> (if not on the same LAN, modprobe netconsole netconsole=+@${IPSRC}/eth0,514@${IPRSYSLOG}/${GWMACADDR} )
<guerby> where GWMACADDR is 00:11:22... your gateway MAC
<guerby> saved the day lots of time here :)
<guerby> found this ext4 bug thanks to it, CVE : https://bugzilla.redhat.com/show_bug.cgi?id=1360968
<guerby> "Red Hat would like to thank Laurent Guerby for bringing this to our attention." and /me thanks netconsole :)
<guerby> https://bugzilla.kernel.org/show_bug.cgi?id=107301
<Ramereth> ya, netconsole is very nice to find odd kernel issues
<h01ger> | and you recommend to have that running all the time on all the nodes? or only when we suspect probs?
<Ramereth> we currently have it running all the time on all physical nodes

3.2. Debian rebuilder thing

(these notes are mostly obsolete now.)

  • prototype only dealing with bullseye and amd64 (and arch all packages)

    • 1st step: download a specific .buildinfo file and sbuild it - done

    • 2nd step: choose a random package from bullseye and fetch the .buildinfo file from builtin-pho and build it - done

      • done: deal with unsigned .buildinfo files, some buildds published a lot of those

      • next: file a wishlist bug for the DIST_BASE= feature…

      • next: file a normal bug for the unclear error messages regarding unavailable build-depends

      • next: file a wishlist for downloading the source and verifying it against the .buildinfo file

      • next: file a normal bug about generating a wrong cmdline for binNMU rebuilds and failing to modify changelog. (block this bug by the download sources feature)

    • 3rd step: put result in db in ionos7: search for NEXT in the _prototype.sh

      • prelimanary db:

        • buildinfofileurl, source, source version, binary version, binary, rebuild_status, rebuildlogurl

    • 4th step: have some scheduler job and a rebuilder job picking up tasks

      • scheduler on ionos7

      • rebuilds on osuosl3

    • 5th step: have a smarter scheduler

    • 6th step: export results as json and html

  • run jenkins job on ionos7 (=buildinfos.d.n), which triggers builds on osuosl3 and stores the results on ionos7

    • keep data in postgresql|sqlite on ionos7

      • ionos7 also has the builtin-pho db so it seems natural to join them…

    • export data as .json via https

    • table with these fields:

      • src-pkg, binary-pkg, sha1 of .deb on ftp.d.o, sha1 of .buildinfo file (signed), sha1 of .buildinfo file (unsigned or signature stripped), signature type, name of .buildinfo file (from ftp.d.o)

      • and either architecture of .deb or filename of .deb (as that includes the architecture)

      • new result: NBIFA - no .buildinfo file available

  • another job to import data on jenkins (into postgresql)

  • schedule/trigger rebuilds on osuosl3, using data in postgresql|sqlite on ionos7

    • scheduler unclear, we’ll first try to rebuild everything in current sid (and bullseye and buster eventually) once, and then, once we’ve done this, we want to do this X times again. (and there will be ftbfs etc, not causing rebuilds immediately.)

    • rebuild amd64 only at first (but arm soon after), but .buildinfo files for all Debian architectures should be tracked in the db

    • also treat the base suite archive and its security, update and backports archives separately

      • store hash of processed Packages file and only process a Packages file if it has an unknown hash → good for suites which are not updated often

      • ignore security archive as those .buildinfo files are not published yet (#862538)

    • good examples: piuparts (arch and any), d-e-i (udeb), d-e-c (all only), emacs (epoch) libunibreak (binNMU)

3.2.1. proper backup

  • FIXME: jenkins.d.n needs to be backed up:

  • /var/lib/jenkins/jobs (the results - the configs are in .git)

  • /var/lib/munin

  • /var/log

  • /root/ (contains etckeeper.git)

  • /var/lib/jenkins/reproducible.db (is backed up manually)

  • /srv/jenkins.debian.net-scm-sync.git (is backed up manually)

  • /var/lib/jenkins/plugins/*.jpi (can be derived from jdn-scm-sync.git)

  • /srv/jenkins.debian.net-scm-sync.git

  • /etc/.git and /etc

3.3. To be done once bugs are fixed

  • #767100 work in progress in etc/munin/plugins/cpu

  • #767018 work in progress in etc/munin/plugins/iostat_ios

3.4. jenkins-job-builder related

3.5. debugging job runs should be made easy

 <      h01ger> | i think the jenkins-debug-job script should be a python script
 <      h01ger> | and j-j-b or another yaml parser can supply job configuration knowledge to that script
 <      h01ger> | \o/
 <      h01ger> | and that python script can also first determine whether the environment is as needed for the job, and if not, complain verbosely+helpfully and exit

4. Improve existing tests

4.1. tests.reproducible-builds.org

4.2. Debian reproducible builds

  • get rid off "set -x # # to debug diffoscoppe/schroot problems"

    • add check if package to be build has been blacklisted since scheduling and abort

    • on SIGTERM, also cleanup on remote build nodes there! (via ssh &)

    • check rbuild logs for "DIFFOSCOPE=E: Failed to change to directory /tmp: Permission denied and deal with those

  • higher prio:

    • reenable disorderfs setup, check that it always unmounts + cleans up nicely

    • pkg pages

      • new table in pkg/test history page: schedule - if that package is currently scheduled

      • add link to pkg set(s) if pkg is member of some

    • link pkg sets and issues, that is: at least show packages without issues on pkg set pages, maybe also some issues which need actions (like uninvestigated test failures)

    • notes related:

      • #786396: classify issue by "toolchain" or "package" fix needed: show bugs which block a bug

      • new page with annotated packages without categorized issues (and probably without bugs as only note content too, else there are too many)

      • new page with packages that have notes with comments (which are often useful / contain solutions / low-hanging fruits for newcomers)

      • new page with notes that doesn’t make sense: a.) packages which are reproducible but should not, packages that build but shouldn’t, etc.

      • new page with packages which are reproducible on one arch and unreproducible on another arch (in the same suite, so unstable only atm)

      • new page with packages which ftbfs on one arch and build fine on another arch (in the same suite, so unstable only atm)

      • new page with packages which ftbfs in testing but build fine on sid

      • new page with packages which are orphaned but have a reproducible usertagged patch

      • new page showing arch all packages which are cross-reproducible, and those which are not

    • new page: "open bugs with patches, sorted by maintainers" page and to the navigation, make those NMUable bugs visible

    • improve ftbfs page: list packages without bugs and notes first

    • bin/_html_indexes.py: bugs = get_bugs() # this variable should not be global, else merely importing _html_indexes always queries UDD

    • once firefox 48 is available: revert 1b4dc1b3191e3623a0eeb7cacef80be1ab71d0a2 / grep for _js and remove it…

  • lesser prio

    • scheduler: check if there have been more than X failures or depwait in the last Y hours and if so unschedule all packages, disable scheduling and send a mail informing us.

    • pkg sets related:

      • add new pkg set: torbrowser-build-depends

      • fix essential set: currently it only has the ones explicitly marked Essential:yes; they and their dependencies make up the full "essential closure set" (sometimes also called pseudo-essential)

      • replace bin/reproducible_installed_on_debian.org with a proper data provider from DSA, e.g. https://salsa.debian.org/dsa-team/mirror/debian.org/blob/master/debian/control

    • a reproducible_log_grep_by_sql.(py|sh) would be nice, to only grep in packages with a certain status (build in the last X days)

    • database issues

      • stats_build table should have package ids, not just src+suite+arch as primary key

      • move "untested" field in stats table too? (as in csv output…)

    • blacklist script should tell if a package was already blacklisted. also proper options should be used…

    • maintenance.sh: delete the history pages once a page has been removed from all suites+archs

    • reproducible.debian.net rename: rgrep all the files…

    • debbindiff2diffoscope rename: do s#dbd#ds#g and s#DBD#DS#g and rename dbd directories?

    • diffoscope needs to be run on the target arch… (or rather: run on a 64bit architecture for 64bit architectures and on 32bit for 32 bit archs), this should probably be doable with a simple i386 chroot on the host (so using qemu-static to run it on armhf should not be needed, probably.)

    • support for arbitrary (to be implemented) Debian-PPAs and external repos, by just giving a source URL

    • once stabilized notification emails should go through the package tracker. The build keyword seems to be the better fit for this. To do so just send the emails to dispatch@tracker.debian.org, setting "X-Distro-Tracker-Package: foo" and "X-Distro-Tracker-Keyword: build". This way people wanting to subscribe to our notification don’t need to ask us and can do that by themselves.

    • repo-comparison: check for binaries without source

    • issues: currently only state of amd64 is shown. it would be better to display packages as unreproducible if they are unreproducible on any architecture.

    • include diffoscope run time in log

  • for _service.sh

    • enabling the service in update_jdn

    • maintenance job might want to:

      • check for running builds using systemctl show & ps fax

      • check if at least one build is running on a build node

    • use ExecStop to kill remote builds on shutdown

      • remove cleanup_nodes.sh once this works

      • add support for disabling archs and for shutdown+respan of workers

      • add support for starting/stopping workers for specific archs

  • missing variations:

    • 32/64 bit kernel variation on i386

    • prebuilder does (user) group variation like this: https://salsa.debian.org/reproducible-builds/reproducible-misc/blob/master/prebuilder/pbuilderhooks/A02_user

    • variation of $TERM and $COLUMN (and maybe $LINES), unset in the first run, set to "linux" and "77" (and maybe "42") in the 2nd run.

      • actually TERM is set to "linux" by default already, COLUMN is unset

    • vary order of $PATH entries, see #844500

    • vary the length of the build paths (/build/first vs /build/second), only once the unreproducibilities caused by different build paths are solved

    • vary the init system: sysv and systemd

    • vary SSD/HDD on i386?

    • vary temp dir variables such as TMP/TMPDIR/TEMP/TEMPDIR/MAGICK_TMPDIR

    • maybe vary build with pbuilder and sbuild (but maybe only useful with different setup jobs only…)

4.2.1. reproducible Debian armhf

  • make systems send mail, use port 465

  • rename all the nodes from $HOSTNAME to $HOSTNAME-armhf-rb ?

    • we could get rid of the links in jenkins.d.n.git/hosts/

    • we could simplify …/hosts/*/etc/munin/munin-node.conf

4.2.2. reproducible Debian arm64

  • vary DEB_BUILD_OPTS? (NUM_CPU)

4.2.3. reproducible Debian installation

4.2.4. reproducible coreboot

  • add more variations: domain+hostname, uid+gid, USER, UTS namespace

  • build the docs?

  • also build with payloads. x86 use seabios as default, arm boards don’t have a default. grub is another payload. and these: bayou coreinfo external filo libpayload nvramcui - and:

    • CONFIG_PAYLOAD_NONE=y

    • CONFIG_PAYLOAD_ELF is not set

    • CONFIG_PAYLOAD_LINUX is not set

    • CONFIG_PAYLOAD_SEABIOS is not set

    • CONFIG_PAYLOAD_FILO is not set

    • CONFIG_PAYLOAD_GRUB2 is not set

    • CONFIG_PAYLOAD_TIANOCORE is not set

  • libreboot ships images, verify those?

  • explain status in plain english

  • use disorderfs for 2nd build

4.2.5. reproducible OpenWrt

  • add credit for logo/artwork

  • explain status in plain english

  • build path variation

  • incorporate popular third-party ("external feeds") packages?

  • html: build variations are wrong

  • html: git commit output includes garbage

  • html: css: add some space on the left side

4.2.6. reproducible NetBSD

  • explain status in plain english

    • explain MKREPRO is set to "yes"

    • explain MKREPRO_TIMESTAMP set to $SOURCE_DATE_EPOCH

  • use disorderfs for 2nd build

4.2.7. reproducible FreeBSD

  • useful improvements:

    • investigate how to use tmpfs on freebsd and build there. see mdmfs(8)

    • find a way to be informed about updates and keep it updated - see freebsd-update cron and pkg audit. The latter is run periodic(8) as part of the nightly root@ emails.

    • modify PATH, uid, gid and USER too and host+domainname as well. The VM is only used for this, so we could change the host+domainname temporaily between builds too.

    • add freebsd vm as node to jenkins and run the script directly there, saves lot of ssh hassle

    • run diffoscope natively

  • TODO: random notes, to be moved to README

    • we build the freebsd master branch

    • we build with sudo too

      • rather not change /usr/obj to be ~jenkins/obj and build with WITH_INSTALL_AS_USER. also not build in /usr/src. if so, we need to define some variable so we can do so…. but we need a stable path anyway, so whats the point.

      • maybe build as user in /usr/src…

  • first build world, later build ports (pkg info…)

  • document how the freebsd build VM was set up:

    • base 10.1 install following https://www.urbas.eu/freebsd-10-and-profitbricks/

    • modified files:

      • /etc/rc.conf

      • /etc/resolv.conf

      • /boot/loader.conf.local

    • pkg install screen git vim sudo munin-node poudriere

      • configure /usr/local/etc/munin/munin-node.conf to allow jenkins to access it

      • configure /usr/local/etc/denyhosts.conf and /etc/hosts.allow and touch /etc/hosts.deniedssh

    • adduser holger

    • adduser jenkins (with bash as default shell)

    • adduser mattia

    • mkdir -p /srv/reproducible-results

    • chown -R jenkins:jenkins /srv/

  • system maintenance

    • upgraded the VM:

      • done with: freebsd-update upgrade -r 10.2 as root in screen

      • and with: freebsd-update upgrade -r 10.3

      • and with: freebsd-update upgrade -r 11.0

      • and with: freebsd-update upgrade -r 11.1

      • and with: freebsd-update upgrade -r 11.2 followed by pkg-static install pkg ; pkg upgrade

      • and with: freebsd-update upgrade -r 12.0 followed by pkg-static install pkg ; pkg upgrade

      • and with: freebsd-update upgrade -r 12.1 followed by pkg-static install pkg ; pkg upgrade

      • and with: freebsd-update upgrade -r 12.2 followed by pkg-static install pkg ; pkg upgrade

      • and with: freebsd-update upgrade -r 13.0 followed by pkg-static install pkg ; pkg upgrade; pkg install screen git vim sudo munin-node poudriere

      • and with: freebsd-update upgrade -r 13.0 followed by pkg-static install pkg ; pkg upgrade

  • online disk resizing howto: https://www.freebsd.org/doc/handbook/disks-growing.html

4.2.8. reproducible Fedora

4.2.9. reproducible Arch Linux

  • setup_archlinux_schroot job:

    • needs to be made idempotent (currently it removes the schroot at the beginning of the job, instead of creating it elsewhere and replacing it on success at the job end…)

    • use schroot tarballs (gzipped), moves are atomic then

  • only disable cert checking on the node running in the future

  • compare the just built pkg.tar.zst with the one available on the arch mirrors. then one can truly say "X% of the Arch Linux packages are reproducible and could bit by bit be reproduced in the real world."

  • maintenance job:

    • check for archlinux schroot sessions which should not be there and delete them. complain if that fails.

  • use db - see https://tests.reproducible-builds.org/reproducibledb.htm

    • scheduler.sh:

      • use asp to update trunk packages?

    • html:

      • leave all pkg.html files, delete them (much) later, then check all pkg.* files are gone

      • recreate them newly as index.html

      • create new job to recreate all pkg pages

      • create json

    • misc:

      • write into all the stats_ tables

      • make build.sh respect pacman exit code

      • comparing versions (in scheduler) is probably needed (though it wuld be better if not): if its not empty it must be higher (due to repo constraints), but it can be even higher than in the repo, because we build trunk

      • debian uses a table removed_packages, should this be used here too?

      • debian uses a table manual_scheduler to limit the amount of packages someone can schedule per day…

      • should breakages job look for archlinux breakages? for which cases? a.) grep "was killed after running into timeout after 30m" pkg.html

    • using notes.git should be next!

  • rebuilding against the archlinux repos

    • only needs to be done once using https://github.com/Foxboron/devtools-repro

    • to not (pacman know about trunk and thus) build trunk, we need to configure asp.

    • we’ll keep building against repo+trunk as we do now (so that archlinux can also benefit from the QA effects)

  • fix build.sh:

    • build2.log doesn’t get deleted if build1 fails

    • → rename build2.log to $version_build2.log (don’t include package name…)

  • things to be done before enabling more builders:

    • build in /srv/workspace instead of /tmp (once this has been done reduce /tmp size back to 15G)

  • build on OSUOSL nodes:

    • update thanks & readme

4.2.10. reproducible qubes

  • add qubes test on t.r-b.o 

        git clone https://github.com/qubesos/qubes-builder
        make get-sources BUILDERCONF=scripts/travis-builder.conf COMPONENTS=installer-qubes-os
        export DIST_DOM0=fc23
        export USE_QUBES_REPO_VERSION=3.2
        export INSTALLER_KICKSTART=/tmp/qubes-installer/conf/travis-iso.ks

        make qubes iso BUILDERCONF=scripts/travis-builder.conf VERBOSE=0 COMPONENTS=installer-qubes-os

  • depends: apt install createrepo python-yaml

  • once this iso is being tested, it will be interesting to build the Qubes templates as well, as those images (Qubes templates are images) will be copied on the installation iso. the above iso is a stripped down iso without templates… (and not the real thing)

4.2.11. reproducible guix

There are two ways to install guix, through prebuilt binaries or through
the source.

*Binary installation:

Go to
<http://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html>,
grab the tarball and follow the instructions there.

*Source instalation:

First make sure you have the dependencies mentioned at
<http://www.gnu.org/software/guix/manual/guix.html#Requirements> installed.

Then download Guix's source from
ftp://alpha.gnu.org/gnu/guix/guix-0.9.0.tar.gz and use the usual
./configure && make && make install

After you have Guix built, you need to create the build-users and have
the guix-daemon run as root, as described here
<https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html>.

Keep in mind that the guix-daemon must always run as root.

*Testing if everything works:

Now just run `guix package -i vim` as a non-root user. If it runs
correctly, Guix is ready for work.

4.3. qa.debian.org*

  • udd-versionskew: explain jobs in README

  • udd-versionskew: also provide arch-relative version numbers in output too

4.4. d-i_manual*

  • d-i_check_jobs.sh: check for removed manuals (but with existing jobs) missing

  • svn:trunk/manual/po triggers the full build, should trigger language specific builds.

  • svn:trunk/manual is all thats needed, not whole svn:trunk

4.5. d-i_build*

  • d-i_check_jobs.sh: check for removed package (but with existing jobs) missing

  • build packages using jenkins-debian-glue and not with the custom scripts used today?

  • run scripts/digress/ ?

4.6. chroot-installation_*

  • use schroot for chroot-installation, stop using plain chroot everywhere

5. Debian Packaging related

This setup should ideally come as a Debian source package, one far away day maybe.