ToDo for jenkins.debian.net

1. About jenkins.debian.net

See "about jenkins.debian.net" for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The the preferred form of contributions is patches via pull requests.

2. Fix user submitted bugs

There are bugs filed against the pseudopackage jenkins.debian.org in the BTS which would be nice to be fixed rather sooner than later, as some people actually cared to file bugs.

3. General ToDo

run all bash scripts with set -eu and set -o pipefail: http://redsymbol.net/articles/unofficial-bash-strict-mode/
- add -o pipefail to all at once first. that should have less fallout that -u.
- though -u is also very nice. it will catch typos.

3.1. nodes at OSUOSL

mention o4+5 in THANKS and explain usage.
mv snapshot.r-b.o from osuosl4 to osuosl5
- setup xfs on o5, then copy snapshot over
- or setup ext4. or read up on xfs.
rebuilder on o4
jenkins backup on o5 (see below)

3.2. 2023 things

rename TODO.trixie to TODO.forky once everything in there has been done
- also s#trixie#forky#g and s#bookworm#trixie#g in there, and check for bullseye, buster and stretch too
setup ionosXY for janitor (remove ionos3 duplicate first)
decouple t.r-b.o from jenkins.d.n (https hosting wise), maybe even www.r-b.o
rename jenkins to hudson? (the hostname)
maintenance job:
- maybe: rm /var/lib/schroot/unpack/d-i-manual* older than 5 days
- maybe: rm /tmp/mmdebstrap.* older than 3 days
djm:
- move fetching logs at the end
- write short djm.README, explain at least djm all d nt and djm --check-setup
- option: --no-fetch (--local? maybe)
- also maybe make --no-fetch default unless overwritten by config? (eg vagrant & mattia hardly ever trigger jobs via UI, while holger does…)
- option: --check-setup to check whether one can login as user to all hosts and su to root (except jenkins where root login is expected)
- option: -r -y ⇒ report for year X
- option: --no-new-xterm or some such
- option: -a/--action (default/implicit/optional), requiring $1 $2 $3 params…
- option: -t/--today to be used with action shell (and maybe others)
- option: --yolo/--dont-wait4-enter, default being wait for enter.
- make special TARGETs . and jenkins implicit for those actions requiring that target only (jenkins-ui, jenkins-restart, etc)
- action: rk / remove-oldest-kernel
- action: sm / shell-monitor
- action: rj / restart-jenkins
- action: mo / mark-offline
- action: so / show-offline
- action: ao / all-online / mark all online
- action: sb|top-builds: 1st stop build service, 2nd stop builds on all nodes
  - only on jenkins
  - as root: service reproducible_build@startup stop
  - as jenkins: /srv/jenkins/bin/reproducible_cleanup_nodes.sh (twice)
- write documentation for djm
- also log an id, so that one command effecting several hosts is counted as on action
  - make actions triggering all be only one entry in the djm logfile → more sensible stats
- maybe:
  - document git commits to jenkins.d.n.git
  - document running "mosh jenkins shell-monitor" processes?
  - document jenkins plugin updates → manually or though my local script which calls the webpages could log this more easily
    
    my local script should call djm and not write logfile entries itself…
  - include notmuch mails, "received and read". write djm-notmuch-parser.
  - include irc. (timeslices when one said something on relevant channels)
- remove "so expect changes for some time." line
- remove/reword "Still very simple statistics"

3.2.1. proper backup

probably do the backup on osuosl5:
jenkins.d.n needs to be backed up:
- /var/lib/jenkins/jobs (the results - the configs are in .git)
- /var/lib/munin
- /var/log
- /root/ (contains etckeeper.git)
- /var/lib/jenkins/reproducible.db (is backed up manually)
- /srv/jenkins.debian.net-scm-sync.git (is backed up manually)
- /var/lib/jenkins/plugins/*.jpi (can be derived from jdn-scm-sync.git)
- /srv/jenkins.debian.net-scm-sync.git
- /etc/.git and /etc