1. About jenkins.debian.net

https://jenkins.debian.net is a tool for automated quality monitoring of Debian. It is work in progress despite being in existence since October 15th 2012.

Get the source by running git clone https://salsa.debian.org/qa/jenkins.debian.net. It’s all in there, no (relevant) manual setup has been done besides what’s in this git repository. (The irrelevant bits are some very simple configuration files containing passwords.)

The (virtualized) hardware is sponsored since October 2012 by https://www.ionos.com (formerly known as Profitbricks) - currently it’s using more than hundred cores and almost 300 GB memory, thanks a lot!

Some stats are available using munin-plugins for jenkins.

Three persons have shell access (incl. root) to the machine: Holger Levsen, Helmut Grohne and Mattia Rizzolo. All of them have also access to the web interface, where tasks like stopping and scheduling job runs can be done, also they have the rights to edit the jenkins scripts (i.e. what jenkins executes) directly, though this is limited to cases like firefighting (IOW deploying changes via the git repository are the norm). The deploying of changes is still limited to people with root powers.

2. Getting involved

jenkins.debian.net is a QA resource for the whole Debian project. Please contact us (via #debian-qa on IRC or via the debian-qa mailinglist) if you / your project is interested to run tests in this setup!

If you notice some jobs has problems and you want to find out why, read debug certain jobs to learn how to do debug jobs locally.

3. Disclaimer

We are happy to take untested patches. Patches are reviewed before merging but no one has a complete test environment for all jobs, so please publish your well meant and untested patches as soon as they are ready!

3.1. Contributing code to this project

It’s helpful to track fixes or new features via wishlist bugs against the jenkins.debian.org package, e.g. with the reportbug tool (devscripts package). The BTS will ensure the developers' mailing list qa-jenkins-dev@lists.debian.org is notified.

The code is available in the jenkins.debian.net GIT repository.

Patches can be submitted as requests to pull from a publicly-visible git repository (this is the preferred way) communicated via IRC or mail, or completely via mail (using git format-patch, see below). If appropriate, please make a topic branch based on the master branch.

You can send patches or requests to the development list, or to the tracking bug: <bugnumber>@bugs.debian.org.

One possible workflow:

  git clone https://salsa.debian.org/qa/jenkins.debian.net
  git checkout origin/master -b <topicname>
  # <edits>
  git commit -a
  git format-patch -M origin/master

  reportbug jenkins.debian.org
  # <describe the issue, attach the patch>

3.2. Contributing bugs to other projects

Another very useful type of contributions is filing bug reports based on jenkins job runs. Another useful kind of contribution would be to improve the documentation, e.g. to better describe how to debug stuff.

4. Notifications

There are two types of notifications being used: email and IRC. At the end of the console log of each build it says to where notifications have been sent. An address of the form jenkins-foo means an IRC notification has been sent to the #foo IRC channel.

All job result notifications should be sent to https://lists.alioth.debian.org/mailman/listinfo/qa-jenkins-scm and optionally to other recipients as well.

5. Jobs being run

There are over 1600 jobs being run currently. If you can think of ways to improve the usefulness of certain jobs, please do give feedback!

5.1. debian-installer jobs

  • d_i_build_$source_package

    • there is one job for each git repo referred to in https://salsa.debian.org/installer-team/d-i/blob/master/.mrconfig

    • each job pdebuilds the master branch of its git repo on every git push in a sid environment. (If the architecture(s) specified in debian/control are not amd64,all or any the build exits cleanly.)

    • while these jobs are triggered on commits, the SCM is only polled every 6min to see if there are new commits.

  • d_i_manual

    • builds the full installation-guide package with pdebuild in sid on every commit to https://salsa.debian.org/intsaller-team/d-i matching suitable patterns.

    • while this job is triggered on commits, the SCM is only polled every 15min to see if there are new commits.

  • d_i_manual_$language_html

    • builds a language (on buster) on every commit of svn/trunk/manual/$LANG with make languages=$LANG architectures=amd64 formats=html.

    • while these jobs are triggered on commits, the SCM is only polled every 15min to see if there are new commits.

    • on successful build, d_i_manual_$lang_pdf is triggered.

  • d_i_parse_build_logs - parses logs from http://d-i.debian.org/daily-images/build-logs.html daily, to give them a bit more exposure.

    • this job is run daily.

5.2. chroot-installation jobs

Installation tests inside chroot environments.

  • chroot-installation_maintenance_$distro:

    • make sure chroots have been cleaned up properly

    • sid jobs are run daily at 04:30 UTC and triggers the $distro specific bootstrap job on success

    • testing jobs are run every 3 days, stable ones once a week, old stable twice a month.

  • $distro-bootstrap jobs:

    • just debootstrap $distro (install a base Debian distribution $distro)

    • there is one job for sid, one for bullseye and one for buster: chroot-installation_sid_bootstrap, chroot-installation_bullseye_bootstrap and chroot-installation_buster_bootstrap

    • on successful run of the bootstrap job, six $distro-install(+upgrade) jobs are triggered.

  • $distro-install jobs (and $distro-install+upgrade jobs):

    • debootstrap $distro, install a $set_of_packages (and upgrade to $2nd_distro)

    • several sets with different packages exist.

      • install is done with apt-get install, except for develop where apt-get build-dep is used to install the build dependencies of these packages.

    • Then there are also all the corresponding upgrade jobs, e.g. chroot-installation_buster_install_gnome_upgrade_to_bullseye

5.3. Debian Edu related jobs

  • All Debian Edu related jobs can be seen at these two URLs:

  • Then there are several types of jobs:

    • tests installation of a profile with preseeding in the graphical installer,

    • screenshots and logs are preserved and a movie created,

    • testing clients against the main-server is planned too, for some time…

      • chroot-installation_$(distro)install$(education-metapackage):

    • tests apt installation of a metapackage in a specific distro.

  • edu-packages_$(distro)_$(src-package):

    • builds one of the six debian-edu packages (debian-edu, debian-edu-config, debian-edu-install, debian-edu-doc, debian-edu-artwork, debian-edu-archive-keyring on every push to its git master branch

    • and whenever debian-edu-doc is built, https://jenkins.debian.net/userContent/debian-edu-doc/ gets updated automatically afterwards too.

5.4. qa.debian.org related jobs

  • There are jobs for lintian and for piuparts:

    • they simply run a build and/or the tests of the master branch of their git repository on every commit against sid. If that succeeds, the same source will be built on testing and stable.

  • There are also jobs related to UDD:

    • they check for multiarch version screws in various suites or issues with orphaned packages without the correct the relevant bug.

  • Last but not least, dpkg related jobs:

5.5. haskell jobs

5.6. rebootstrap jobs

5.7. reproducible builds jobs

  • See https://wiki.debian.org/ReproducibleBuilds to learn more about "Reproducible Builds" in Debian and beyond.

  • Several jobs are being used to assemble the website https://tests.reproducible-builds.org which is actually a collection of static html and log files (and very few images) being served from this host. Besides the logfiles data is stored in a database (schema description at https://tests.reproducible-builds.org/reproducibledb.html) which can be downloaded from https://tests.reproducible-builds.org/reproducible.sql.xz. (That copy is updated daily.)

  • The (current) purpose of https://tests.reproducible-builds.org is to show the potential of reproducible builds for Debian - and six other projects currently. This is research, showing what could (and should) be done… check https://wiki.debian.org/ReproducibleBuilds for the real status of the project for Debian!

  • For Debian, six suites, buster, bullseye, bookworm, trixie, unstable and experimental, are tested on four architectures: amd64, i386, arm64 and armhf. The tests are done using pbuilder through several concurrent workers: 40 for amd64, 24 for i386, 32 for arm64 and 35 for armhf, which are each constantly testing packages and saving the results of these tests. There’s a single systemd service starting all of these workers which in turn launch the actual build script. (So the actual builds and tests are happening outside the jenkins service.)

    • To shutdown all the workers use: sudo systemctl stop reproducible_build ; /srv/jenkins/bin/reproducible_cleanup_nodes.sh

    • To start all the workers use: sudo systemctl start reproducible_build

  • These builds on remote nodes run on very different hardware:

    • for amd64 we are using four virtual machines, ionos(1+5+11+15)-amd64, which have 15 or 16 cores and 48gb ram each. These nodes are sponsored by IONOS.

    • for i386 we are also using four virtual machines, ionos(2+6+12+16)-i386, which have 10 or 9 cores and 36gb ram each. ionos2+12 run emulated AMD Opteron CPUs and ionos6+16 Intel Xeon CPUs. These nodes are also sponsored by IONOS.

    • for arm64 we are using eight arm64 kvm nodes, which have 12 cores and 32gb ram each. These nodes are sponsored by Codethink.

    • To test armhf we are using 15 small boards hosted by vagrant@reproducible-builds.org:

      • two quad-cores (cbxi4pro0, wbq0) with 2gb ram,

      • three quad-cores (cbxi4a, cbxi4b, ff4a) with 4gb ram,

      • one hexa-core (ff64a) with 4gb ram,

      • four quad-core (virt32b, virt32c, virt64b, virt64c) with 7gb ram,

      • four quad-core (virt32a, virt32z, virt64a, virt64z) with 15gb ram,

  • We would love to have more or more powerful ARM hardware in the future, if you can help, please talk to us!

  • Packages to be build are scheduled in the database via a scheduler job, which runs every hour and if the queue is below a certain threshold schedules four types of packages:

    • new untested packages (either uploaded to unstable or experimental or migrated to trixie, or security updates to bookworm, bullseye or buster),

    • new versions of existing packages, which were already tested - these are always scheduled, no matter how full the queue is

    • old versions, already tested (at least two weeks ago)

    • and also some old versions which failed to build (at least ten days ago), if no bug has been filed.

  • To manually schedule a package build, you need a Debian SSO SSL client certificate. Then you can navigate to each package page (i.e. https://tests.reproducible-builds.org/debian/linux) and click on the recycling icon ♻. The called CGI script can do more with special options, but those feature are not currently documented.

    • If you do not have a Debian SSO certificate or if you want to reschedule big sets of packages please ask for a manual rescheduling in the #debian-reproducible IRC channel on OFTC. Those with shell access to jenkins can bypass the limitations imposed to remote calls, which are limited to 500 schedulings per day, which should be plenty for normal usage.

  • Several other jobs exist to build the HTML pages and to create two JSON files which can be downloaded from https://tests.reproducible-builds.org/reproducible.json and https://tests.reproducible-builds.org/reproducible-tracker.json. The 1st one has all the data (except history) and the 2nd has all the data we consider relevant to bother maintainers with, that is, some ftbfs isses are excluded.

  • Information from https://salsa.debian.org/reproducible-builds/reproducible-notes is incorporated on pushes to that git repo.

  • There are suite specific jobs to create the pbuilder base.tgz’s per suite, which have the reproducible apt repo added. Similarly there’s another job per suite to create the schroots used by the builder jobs to download the packages sources to build.

  • Then there are two more jobs to create sid and testing schroots to run diffoscope on the two results. This is necessary since to investigate haskell binaries, diffoscope needs access to the same haskell compiler version as the investigated packages have been built with.

  • For making sure things are considerably under control at any time, there is a maintenance job running every 3h, mostly doing cleanups.

  • The jenkins job overview at https://jenkins.debian.net/view/reproducible/ probably makes it clearer how the job scheduling works in practice.

  • Blacklisting packages from building can be done by those with shell access (as user "jenkins"): 

jenkins@jenkins:~$ /srv/jenkins/bin/reproducible_blacklist.sh $arch $suite $package1 $package2 ...

  • We support sending automatic email notification for status changes to maintainers. Enabling/disabling these notifications can be done by people with shell access to jenkins: 

jenkins@jenkins:~$ /srv/jenkins/bin/reproducible_setup_notify.py -h
usage: reproducible_setup_notify.py [-h] [-o] [-p PACKAGES [PACKAGES ...]]
                                    [-m MAINTAINER]
  -h, --help            show this help message and exit
  -o, --deactivate      Deactivate the notifications
  -p PACKAGES [PACKAGES ...], --packages PACKAGES [PACKAGES ...]
                        list of packages for which activate notifications
  -m MAINTAINER, --maintainer MAINTAINER
                        email address of a maintainer

5.8. jenkins.d.n jobs

These are jobs for making sure jenkins.debian.net is running smoothly.

6. Debugging certain jobs

To debug most jobs, a jenkins setup is actually not needed.

  • In principle the shell commands from the various jobs should run on any Debian system just fine. Please use a test system though, as all your data might be eaten.

    • A good first step is to use this git repo as a Debian source package, build it and then install the jenkins.d.n-debug package and all it’s recommends on your test system. NOTE: this ain’t as helpful as it used to be as many depends have only been added to update_jdn.sh and not to debian/control.

6.1. Feedback

We love to get feedback on this! Either by sending an email to debian-qa@lists.debian.org or by joining #debian-qa on irc.debian.org and expressing yourself there. The best way is to report bugs, even better if accompanied by patches or pull requests. But really, all feedback is appreciated!

6.2. Setup

See INSTALL.

6.3. ToDo

There is still a lot of work left, check the current ToDo list.

6.4. Thanks

See THANKS.

7. License