1. About jenkins.debian.net

See "about jenkins.debian.net".

2. Setup

2.1. Installed software that makes jenkins.debian.net run

jenkins.debian.net runs Debian Buster.

2.2. Jenkins

Currently running the long term support .deb from jenkins-ci.org.

Jenkins is installed using deb http://pkg.jenkins-ci.org/debian-stable binary/ into /etc/apt/sources.list.d/jenkins.list and apt-get update && apt-get install jenkins.

2.2.1. Jenkins plugins

  • Git plugin

  • Subversion Plug-in

  • Email Extension Plugin

  • Log Parser Plugin

  • HTML Publisher plugin

  • Read-only configurations

  • SCM Sync Configuration Plugin

  • Dependency Graph Viewer Plugin

  • View Job Filters

  • Console Column Plugin

  • Priority Sorter Plugin

  • Throttle Concurrent Builds Plugin

  • Build Blocker Plugin

  • Naginator Plugin

  • Next Executions

  • Parameterized Trigger plugin

  • Environment Injector Plugin

  • build timeout plugin

  • Build Authorization Token Root Plugin

  • Embeddable Build Status Plugin

  • Plot Plugin

  • GitLab Plugin

  • Sidebar Link

  • Green Balls

  • Compact Columns

  • Console Tail Plugin

  • Cron Column Plugin

  • Simple Theme Plugin

  • Live Screenshot

  • Locale plugin

2.3. updating jenkins.debian.net with update_jdn.sh

update_jdn.sh is a hackish script which will install all needed packages, copy their configuration in place, generate the documentation webpages and update all jenkins jobs if needed.

As user run:

cd
git clone https://salsa.debian.org/qa/jenkins.debian.net
cd jenkins.debian.net
./update_jdn.sh

vncdotool is the only software which is not installed by packages and is not covered by this script.

The configuration of the openQA secrets is done manually. The file /etc/openqa/client.conf (on osuosl3) contains the API key for accessing the openQA server.

  • In Jenkins, go to Dashboard | Credentials | System | Global credentials

  • Add 2 entries (one for ID=openqa-live-apikey and one for ID=openqa-live-apisecret)

  • Kind: Secret Text

  • Scope: Global

  • Secret: see client.conf

  • ID: openqa-live-apikey

  • Description: The apikey from /etc/openqa/client.conf on osuosl3

2.4. IRC notification configuration using kgb-client

The jenkins jobs are configured to send email to jenkins+$IRC_CHANNEL (like jenkins+debian-qa), this is parsed by a script (/srv/jenkins/bin/email2irc.sh) run through procmail which then in turn notifies kgb-client, which notfies kgb-server(s) on the internet, which are the bots notifying the IRC channels.

The Jenkins Email notification plugin is used as its state tracking is best (and the Jenkins IRC plugin is way too powerful).

2.5. Installing vncdotool

apt-get install python-twisted python-imaging
git clone https://github.com/sibson/vncdotool
cd vncdotool/
python setup.py install

2.6. Adding new agents to jenkins

Process to follow to add a new node to jenkins:

  • Open https://jenkins.debian.net/computer/new

  • Use the FQDN of the new node, and select "Permanent Agent"

  • Configure it as follow:

  • # of executors: the number of CPUs of the new node

  • Remote root directory: /srv/jenkins/pseudo-hosts/$hostname

  • Labels: $arch $short_name (with $short_name like ff64a if the hostname is ff64a-armhf-rb)

  • Usage: select "Only build jobs with label expressions matching this node"

  • Launch method: select "Launch agent via execution of command on the master"

  • Launch command: /srv/jenkins/bin/start-agent.sh

  • Availability: select "Take this agent online when in demand, and offline when idle"

  • In demand delay: 0 (so that builds will start right away)

  • Idle delay: 5 (this is an arbitrary amount of time)

The agent setup is done so that the agent.jar program doesn’t get run on the remote nodes, to avoid needing Java available in there. Jobs wanting to run entirely on a remote host will just execute jenkins_master_wrapper.sh which will do the needed ssh and execute jenkins_node_wrapper.sh on the remote host, which will need to know what to execute after to actually start the wanted program.

2.7. munin

Munin is installed, running with jenkins plugins not yet published elsewhere.

2.8. Using jenkins-job-builder

update_jdn.sh takes care of this too, so this is just a quick run through. Make sure to also look at the upstream documentation at https://jenkins-job-builder.readthedocs.io/en/latest/

  • run cd /srv/jenkins/job-cfg ; jenkins-job-builder update . to update jobs

  • for testing: cd /srv/jenkins/job-cfg ; rm test-output/ -r ; mkdir test-output ; jenkins-job-builder test . -o test-output/ ; ls test-output/

2.9. Backup

Currently only the configuration is backed up, but neither the jenkins results nor system logs. But that’s at least sufficient to re-setup the system if needed.

  • The etckeeper package is installed and automatically keeps /etc in /etc/.git.

  • The SCM Sync Configuration Plugin is installed in Jenkins, putting all Jenkins configuration into /srv/jenkins.debian.net-scm-sync.git.

    • (This is not ideal yet, because it also puts the generated job configuration there…)

I’ve cloned these two git repos on my desktop and manually run git pull. This is certainly not ideal but good enough to re-setup the system anytime, which is something.