Jenkins maintenance

Jenkins is the tool we picked for our automated tasks.

Jenkins server

We have a master server, currently provided by Theke Solutions. It has a regular Jenkins install using Debian packages (on Ubuntu 16.04). It is using the Jenkins LTS distribution. This server only runs the Jenkins app, no tests are run there.

Jenkins nodes

Jenkins nodes are VMs that have a jenkins user created, and set with an SSH pubkey we have for that. They have Java 8 on them, the Jenkins server needs the java command available on the nodes to launch its shell. Since we containerized the runs using Docker, the jenkins user is required to belong to the docker group on the host machine.

The nodes are provided by different entities.:

Docker1: ByWater Solutions
Docker2: Universidad Nacional de Córdoba
Docker3: KohaAloha

Jenkins tasks

The Jenkins tasks are basically the tasks we run on the nodes. All of them are triggered by (1) polling on some repository (running the full test suite for Koha for example) or (2) by calling them via webhooks (koha-testing-docker builds triggered from gitlab on each commit).

Koha tests

A while back when koha-testing-docker was created, it targeted the Jenkins use case. i.e. it was designed to run the whole test suite and exit. koha-testing-docker evolved into a KohaDevBox replacement, but there's still a RUN_TESTS_AND_EXIT flag that triggers that original behaviour.

Koha testing tasks basically:

Set some environment variables:

$ export RUN_TESTS_AND_EXIT="yes" # mentioned before
$ export SYNC_REPO=.   # use the 'current' checkout as SYNC_REPO
$ export LOCAL_USER_ID=$(id -u)   # run as the calling user
$ export COVERAGE=$COVERAGE   # calculate tests coverage? off by default because of the x4 running time, should be run on a schedule, maybe weekends
$ export KOHA_IMAGE=master-jessie # what koha-testing-docker image to use

Download the docker-compose.yml and env files, and run docker-compose.
Perform some cleanup tasks before and after the tests are run. We faced many issues with Docker filling the disks with old stuffs from each run, and some abnormal terminations leaving running containers/networks and preventing new runs.

Tasks for a new release

When a new release needs to be set on Jenkins, there are a few easy things to consider/do:

As the tasks per-se got really generic (i.e. 'run this koha-testing-docker image against this git checkout) when a new release needs it, we use the 'Copy from' option in the 'New task' form. So we copy the task for 'master', using the OS we want. After creating the (copied) task, we just change the branch the task will be polling.

CAVEAT: The Koha repo is huge. So cloning it takes loads of time and it always timeouts on jenkins after 10 minutes or so. It also costs money (inbound traffic). So what we do (for now) is having a full kohaclone in /srv/jenkins/workspace/Koha_Source, and create symlinks. This is the nastiest bit I do in Jenkins:

$ cd /srv/jenkins/workspace
$ ln -s Koha_Source Koha_19.05_D8
$ ln -s Koha_Source Koha_19.05_D9
$ ln -s Koha_Source Koha_19.05_U18

This needs to be done on each Jenkins node. To avoid races on the Koha_Source repository, runs are restricted to one running task on each node. This is also a constraint we have because of the fact that koha-testing-docker will load Elasticsearch and Selenium. Running more than one task proved counter productive (system load skyrockets, memory is not enough, etc).

koha-testing-docker builds

koha-testing-docker (master) project has been recently refactored so the three Dockerfiles (for Jessie, Stretch and Bionic) are on a single branch, to ease maintainability. The shipped Jenkinsfile on the koha-testing-docker project has been adjusted to build all images in one run. Before that we had a branch and a task for each. This builds are of type 'Pipeline' and so they are controlled by the Jenkinsfile included on the repository.

Job configuration recommendations

delete workspace before build starts
Source Code Management: honor refspec on initial clone
Source Code Management: Path of the reference repo to use during clone: /srv/jenkins/workspace/Koha_Source
Source Code Management: shallow clone
post-build actions: delete workspace when build is done