Continuous Integration

From Koha Wiki
Jump to navigation Jump to search

Continuous integration in Koha

Continuous Integration is when you automatically build things (unit tests, documentation or whatever else) each time a commit is done and the system provides notifications for each fail.

In Koha the tool used as CI server is Jenkins

Some notes about the Jenkins interface:

  • On the start page each line represents a job with its characteristics: name, git branch built, scripts ran, etc.
  • Click on Koha_Main in the left hand navigation: http://jenkins.koha-community.org/job/Koha_main/
  • Build history on the left shows failed (red) and successful builds (red).
  • Each time a build fails, a notification by email is sent to RM and the person who breaks the build - doesn't appear to be active right now
  • HTML_Report gives the cover of tests by file / sub - link broken

Config of Jenkins

Jenkins fires up a koha-testing-docker (ktd) container that will run a command to run some tests.

A job is triggered when a new change is made to the git repository. Most of our jobs are tracking the Koha gitlab repository. This repository is checked every 5 minutes. However the gitlab repository is configured as a mirror of our official repository and we do not control the delay for synching with gitlab.

All of that is to say that Jenkins' jobs are not triggered right after a push is done.

Type of job

There are two kinds of jobs:

  • full: The full job runs "all tests".
  • light: The light job runs only the usual Perl tests.

The projects Koha_Main, Koha_23.05_D11, etc are defined to do a full run, while others will only do a light run. D11 was picked because it's stable now.

The projects suffixed with _ES6, _ES7, _ES8, _OS1 and _OS2 run the Elasticsearch/OpenSearch tests with different versions of the search engine.

Type of tests

We have:

  • Selenium tests
  • Cypress tests
  • Elasticsearch tests
  • Other tests (mostly xt and t, without Elasticsearch and Selenium tests)

The light run will only run the "other tests", while the full runs will run all of them AND the database update process.

A typical full run will:

  • Drop the database
  • Run the onboarding test: selenium/00-onboarding.t
  • Drop the database
  • Run the update database process, twice
  • Inject a v19.11.00 database dump
  • Run updatedatabase.pl
  • Reset Koha version to 21.11.00
  • Run updatedatabase.pl, to ensure idempotency
  • Drop the database
  • Inject the sample data
  • Run selenium tests (not 00-onboarding, first 01-installation.t that will select all the sample data, then the other selenium tests)
  • Run all the other perl tests in a random order
  • Run Cypress tests, but only if all other tests are green!

A light run will:

  • Run all the "other tests" in a random order, using the existing database

Build step

When a job is triggered, Jenkins will enqueue it and when a node is available the following script will be run.

For a full run

#!/bin/bash -x
export SYNC_REPO=.
export KOHA_IMAGE=main
export TEST_SUITE=full # run all the test suite (including ES)
export KTD_BRANCH=main

wget -O docker-compose.mariadb_d11.yml https://gitlab.com/koha-community/koha-testing-docker/raw/$KTD_BRANCH/docker-compose.mariadb_d11.yml
export DBMS_YML=docker-compose.mariadb_d11.yml

wget -O build.pl https://gitlab.com/koha-community/koha-testing-docker/-/raw/$KTD_BRANCH/jenkins_config/build.pl
/usr/bin/perl build.pl

For a light run

#!/bin/bash -x

export SYNC_REPO=.
export KOHA_IMAGE=main-bookworm
export TEST_SUITE=light

wget -O docker-compose.mariadb_d12.yml https://gitlab.com/koha-community/koha-testing-docker/raw/main/docker-compose.mariadb_d12.yml export DBMS_YML=docker-compose.mariadb_d12.yml

wget -O build.pl https://gitlab.com/koha-community/koha-testing-docker/-/raw/main/jenkins_config/build.pl
/usr/bin/perl build.pl

What's happening? We retrieve the docker compose files, the jenkins_config/build.pl files from the corresponding version of ktd, then run it.

jenkins_config/build.pl will simply:

  • pull new docker images
  • start ktd with RUN_TESTS_AND_EXIT set to "yes"
  • so that the container will run the test suite
  • exit the container
  • clean up (docker prune)

When RUN_TESTS_AND_EXIT is set to "yes", misc4dev/run_tests.pl will be run with the relevant options, depending on the type of the run (`--run-all-tests`, `--run-light-test-suite`, `--run-selenium-tests-only`).

Other options for this script are available.

Statuses on the dashboard

On the Koha dashboard there is a section that shows all our builds and their status (passed, unstable, failing).

Coverage

The full runs can be scheduled with the COVERAGE flag set to 1. It will run the whole test suite with Devel::Cover that will generate statistics about the code coverage.

We usually run one at the beginning of a new release cycle, and keep the result, so that we have a graph on the page of the project. Example: Koha_Main

The nodes

There are currently 7 Jenkins nodes. Only 3 are big enough to run the full runs (mainly because of the RAM needed by Elasticsearch): Docker_3, Docker_4 and Docker_8.

  • Docker_1 is hosted by Universidad Nacional de Córdoba
  • Docker_2 by Universidad Nacional de Córdoba
  • Docker_3 by Mason (online from 10pm to 7am NZDT)
  • Docker_4 by Universidad Nacional de Córdoba
  • Docker_5 by University of Helsinki
  • Docker_7 by Orex (administered by Joubu, same server as splitter2)
  • Docker_8 by PTFS-E

How to make Jenkins happy?

If you are a developer

When you get a new patch pushed, you are responsible for the failures your patches introduce. Please keep an eye on the next build and if a failure happens, take care of it, if you caused it.

If you are RM/RMaint

If the build triggered by one of your pushes is failing, you must not push anything before you have fixed the problem!

How to fix a regression?

  • Try to find a commit where the problem didn't exist.
  • You can easily find one on Jenkins.
  • Look at the failing build id -1.
  • For instance if Koha_Main/42/ is failing, look at Koha_Main/41/ and notice the "Revision: " line: it contains the commit ID on which the build has been run.
  • Then use git bisect locally: `git bisect start HEAD BAD_COMMIT_ID` and use `git bisect good` or `git bisect bad` until git finds the culprit.
    • If it's from you, fix it! Attach the patch to the bug report and add the "additional_work_needed" Bugzilla keyword. This will ensure that people are alerted to your fix.
    • If it's from someone else, feel free to fix it, otherwise leave a note on the bug and also add the "additional_work_needed" keyword.
    • To be in the same state as Jenkins you can use the /kohadevbox/misc4dev/run_tests.pl script that is available in the koha-testing-docker's container.

Bisecting

When you are trying to find a commit that broke some test(s), you usually need to `reset_all` between steps. And then run the tests to verify: 

git bisect start HEAD BAD_COMMIT_ID
reset_all
prove t/db_dependent/Failing/Test.t && git bisect good || git bisect bad

Luckily, `KTD` ships an alias that will automate this: 

git bisect start HEAD BAD_COMMIT_ID
bisect_tests t/db_dependent/Failing/Test.t

Run automated tests

For instance:

Run everything

  • Warning: It will drop your database!
    perl /kohadevbox/misc4dev/run_tests.pl --run-all-tests
    Run Cypress tests only
    perl /kohadevbox/misc4dev/run_tests.pl --run-cypress-tests-only
    Run Selenium tests only
    perl /kohadevbox/misc4dev/run_tests.pl --run-selenium-tests-only

Get notified

This channel on Mattermost https://chat.koha-community.org/koha-community/channels/ci-cd-notifications will notify you when jobs succeed or fail. You must join it!

Overview

For a summary, a graph:

Architecture of a Jenkins build



Developer handbook

Retrieved from "https://wiki.koha-community.org/w/index.php?title=Continuous_Integration&oldid=34919"