Release maintenance
Note: this guidelines are work in progress, and release maintainers have the right to change this to whatever they prefer. For now this will serve as a documentation place for the tasks involved in release maintenance.
The Release Maintainer role
Release maintainers do the task of keeping the stable Koha releases in line with patches provided for bugfixes. People are elected for this task after they have volunteered for it, and there is consensus on the development community for them to take on the job.
RMaints are assigned one or more stable branches to maintain. Crazy people like Chris Cormack, Chris Nighswonger and Fridolin Somers have maintained more than one release at a time. It's a crazy Chris thing®.
Schedule
Every month there is a stable maintainance release. The relevant dates are:
- 15th: String Freeze
- 22nd: Release
String Freeze: During String Freeze patches that change strings should not be pushed! Please keep in mind that a changed string will mean that the string appears untranslated after updating, which can be especially toublesome for OPAC strings. Most users rely on the packages to provide translations and are not aware how they could fix such problems locally.
- Announce String Freeze on the koha-translate mailing list and user list (koha@lists.katipo.co.nz)
Release: After each release, get in touch with your fellow release maintainers and the packaging manager in order to agree about a date for string freeze, a date for when your branches will all be ready for packaging and a date for the release. This is thought to make the life of our packaging manager easier as they can build all branches at once. Also it means that the packages will be ready at release.
Initial setup
Request push access for git
You will need to be given privileged access to the community git server and it's security counterpart.
To enable this, you will need to get access to https://git.koha-community.org/Koha-community/ (ask an account on IRC or koha-devel mailing-list) and link your public ssh key to your gitea account.
Create a remote for pushing
You could set up a remote for pushing:
git remote add upstream git@git.koha-community.org:Koha-community/Koha.git
and then push using this:
git push upstream 18.11.x
Git pre-push hook
There is a pre-push hook documented in the 'Tips & Tricks' page, this can be beneficial as it adds a series of final checks before allowing you to push your local updates to the public repository.
Satisfied users testimonials:
- «I'm not worried about pushing to the wrong branch anymore!»
- «The pre-push hook detected a forgotten SCSS compilation since many patches»
- «I would have pushed so many commits without a signoff without this hook»
- «Helps to not mix things up when releasing two branches»
Release tools
This section lists the set of tools a release maintainer needs to be able to make releases.
The release-tools repo
The Koha dev community has a nice set of tools to make the job easier. This set of scripts is on a git repo that can be cloned using
$ git clone https://gitlab.com/koha-community/release-tools.git
GPG keys
In order to sign the git version tag you need to have valid gpg signatures. Generate them issuing:
$ gpg --gen-key
Keys can be generated using 4096 or 2048 bits. It took me a while and lots of USB HDD activity to get enough entropy but I managed to generate it with 4096 bits.
You may need to configure your git repo in order to sign git commits with your GPG key.
You can add the key quickly to a repo (or all repos) with
$ git config --global user.signingkey [gpg-key-id]
Request access to the community website
It will be used to announce the new releases.
For getting access on services in general, see this page to find who to ask:
Handover between two cycles
The old and new release maintainer of a given branch should contact each other to share this information:
- The position of the current "RMaint cursor" i.e. the last bug considered for backporting.
- The eventual list of bugs that due to the last string freeze have not been yet backported.
- The eventual list of bugs that had a backport request but are too hard to backport and are waiting a rebase to be unblocked.
- Any other significant relevant info like gotchas of the branch (in conflicts, in random automated tests failures, etc).
Daily maintenance workflow
Release Maintainers usually watch the activity on the main git branch. If they maintain a release other than the latest stable release, they should be looking at higher-numbered versions git logs (for example, if you are maintaining the 22.11.x stable branch, then you should be probably watching 23.05.x, and keep in touch with its maintainer).
Important: Sometimes there will be bugs specific to your version, something that has been missed previously or otherwise requires attention from a specific RMaint. These are marked in bugzilla with the rel_XX_XX_candidate keyword for your version. You should make it part of your daily routine to check for those bugs. Once resolved, the keyword can be removed.
MRenvoize uses a quick and dirty script to add bugs from main into hist todoist list.
You should also watch for automatic debian/control updates for your specific branches:
Once you spot a specific commit or commit series on your queue, the first thing is to look at bugzilla and check if it is a bugfix or an enhancement and decide whether to back port.
Deciding on what to back port
The general principle is to maintain stability and don't break things for supported releases!
| Release | Aim for the release | Guidance for back porting |
|---|---|---|
| all releases | Stable and reliable |
|
| stable | Most feature-rich release, there may be some bugs |
|
| oldstable | Tried and tested, more bug-free |
|
| oldoldstable | The most bug-free release |
|
Other notes:
- Important notes: if the tag 'important' is used on a comment, it will stand out in red and also create a note on top of the bug with the number of important entries. Please make sure to check these as they might contain important information targeted at RMaints.
- Enhancements: Generally only back port minor enhancements, leave a message on the bug to say it can be done on demand if required.
- Seek clarification where required: Check with the rest of the devs on IRC or by mailing the author of the patches for clarification.
- Changes with database updates: Be particularly cautious about back porting a change that includes a database update. All database updates 'should' be idempotent, meaning they can be run repeatedly without negative consequences. It's always worth double checking this as errors have been known to creep through occasionally. For example, a DB update that adds a new table will result in a warning, at least, during upgrade, and should be avoided unless it is absolutely required in order to resolve a security bug. When it doubt, please discuss with the RM and fellow Release Maintainers first.
When not back porting
Mark the bug as either "Needs documenting" or "Resolved - Fixed" if you are happy to judge the bug as not requiring any further documentation changes.
It's also nice to leave a comment if relevant along the lines "Not back porting, not relevant to VERSION" or "Not back porting, new feature" etc
Processing a commit or commit series
Once you have decided the patch should be back ported to the stable branch you are maintaining, update your local git repository (just in case) and create a working branch. You should define a naming schema to avoid confusion now you have commit rights! As 18.11.x release maintainer I chose 18.11bug_<bugnumber>. Do what you want :-D
git checkout upstream/18.11.x -b 18.11bug_<bugnumber>
You should then look for the commit hash of the patch(es). And then cherry-pick them like this:
git cherry-pick -x 1a8db0ba7d62a70cd6b49ee705a5db9b11ad7672
If there's some conflict with cherry-picking, git will tell you. You will have two options
- Solve the conflict
- Try to solve the conflict and give up
If solving the conflict is not straightforward, comment on the bug like this: 'This does not apply cleanly to 18.11.x please submit one formatted for 18.11.x'.
If you solved the conflict you finish by
git add <touched files> git commit
A cherry-pick message is added to the commit, and if there were conflicts, on which files.
Note: If the patch set includes any database updates, you will need to correct the version number bump to match your branch and running number.
Note: If the patch set includes changes to database table structures you will likely need to rebuild the DBIx::Class schema files. See Release management/Update the schema files
Note: If the patch set includes a database atomic update, like if it's only for your branch, see Database updates/Pushing to a branch
Note: If the patch set includes CSS changes, you will likely need to rebuild the CSS files from their source SCSS for testing. See: Working with SCSS in the OPAC and staff interface
Note: You should run some tests. For example, if the patch modifies several templates, you should at least run:
perl xt/tt_valid.t perl xt/author/valid-templates.t
Do some testing, and keep in mind you should be confident about the QA team.
Once everything is solved, and testing proves everything is fine with the cherry-picked commits, you should sign on them:
git commit --amend -s
And then merge the changes with your maintenance branch
git checkout 18.11.x git merge 18.11bug_<bugnumber> --ff-only
Pushing upstream
Once you've collected some bugs in your local branch, you need to remember to push it up to the community.
git push upstream 18.11.x
Assuming you have configured the git pre-push hook, using this branch should run a final pre-push check on your work before allowing you to push your changes up to the wider world.
Update Bugzilla
This is very important: Let people know you pushed the patches to your stable branch by changing the status to Pushed to stable/oldstable/oldoldstable and we also now have a custom field (Version(s) released in:) in Bugzilla which you should add the specific release version to for which the bug will be released (This field takes the form of a comma-delimited list of versions, for example, `19.05.00, 18.11.03, 18.05.11`).
Note: You could use the `koha-push` script from the release-tools repository to do the bugzilla update. It will do a final check on the bug status, update the status, add text to 'Version(s) released in' and add a comment to the bug for you.
Monitoring Jenkins
We use Jenkins, a CI tool, to test each branch after new code is pushed to it. You should have been added to the email contacts for failing jobs and as such should get notified of any test breakages shortly after pushing any changes.
You can view the progress of the build tests on the Koha community Jenkins site. For the 18.11 branch, this takes about 30 minutes for each build.
Releasing
Push updated translation files [skip]
Since 2023 November releases the po files are no longer part of the Koha source code. They will be installed using the koha-l10n debian package.
You don't need to do anything!
Ensure the installer and upgrade database haven't fallen out of sync
git checkout v22.11.last_release
reset_all
git checkout 22.11.x
updatedatabase
perl misc/devel/update_dbix_class_files.pl --koha-conf $KOHA_CONF
git commit -a -m"test diff"
dbic
git diff
Use the diff to work out if anything needs to be changed in either the db_revs included in this cycle or the kohastructure file. Make any changes required and commit them.
Remember to clean up the 'test diff' commit created above (rebase and drop that commit)
Bump version number
In Koha.pm increment the following line:
$VERSION = "YY.MM.XX.000";
Remember that between releases there might be minor database version upgrades. They would have been done the same way, using the latest minor number to reflect those version changes. For example 17.05.02.001, 17.05.02.002 and so on.
Create a new db_rev file in /installer/data/mysql/db_revs with name "211110000.pl" (for 21.11.10.000)
use Modern::Perl;
return {
bug_number => undef,
description => 'Koha 21.11.10 release',
up => sub {},
}
You should then commit those changes:
$ git add Koha.pm /installer/data/mysql/db_revs/211110000.pl $ git commit -m 'Increment version for YY.MM.XX release' $ git commit --amend -s
You can now push the latest changes. Or continue with the next step.
Generate the release notes
NOTE: Make sure to pull the latest version of the release tools before generating the notes
Release notes are automatically generated by the koha-release script from the release-tools repository. If you are releasing version 22.11.03 you should run this on your clean 22.11.x branch:
$ ~/<path_to>/release-tools/bin/koha-release v22.11.02..HEAD notes # The HTML version is generated from the Markdown one. # So if editions need to be made, they can be done to the Markdown version only and the HTML can be regenerated. $ ~/<path_to>/release-tools/bin/koha-release v22.11.02..HEAD html misc/release_notes/release_notes_22_11_03.html
it will create the following files:
- misc/release_notes/release_notes_22_11_03.md - misc/release_notes/release_notes_22_11_03.html
Check for inconsistencies
It is important that you edit that file and fix any inconsistencies you find. Usually:
- First release: The team might be wrong if the config file has the wrong main version. (release-tools/etc/user.yaml)
- Security release: security patches aren't listed if etc/user.yaml doesn't contain the Bugzilla credentials. As the tickets are private.
- The branch name might be wrong at the end of the file. It will likely be wrong when doing a security release (e.g. fix it with 20.05.x instead of 20.05.x-security)
- If there are no features/enhancements/bugfixes the current script may generate empty sections for them.
- It should point to System requirements and recommendations.
Just make sure everything is sound. Once you are done, you should commit the release notes:
$ git add misc/release_notes $ git commit -m 'Update release notes for YY.MM.XX release' $ git commit --amend -s
It's not a bad idea at this point to push your branch upstream one last time and watching all the tests pass before tagging it for release.
Tag the release
Tagging will prompt for your GPG passphrase. It is done issuing:
$ git tag -a v22.11.03 -s -m 'Koha release 22.11.03'
Note the two digits. You can now push the new tag doing:
$ git push upstream v22.11.03
Since November 2023 the tarball will be generated when a new tag is pushed. The git push command of the tag will take a while: the generation of the tarball will happen in the background.
You should see the following output (Without the comment lines):
# Push of the tag to local and github mirrors Enumerating objects: 1, done. Counting objects: 100% (1/1), done. Writing objects: 100% (1/1), 192 bytes | 192.00 KiB/s, done. Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 remote: . Processing 1 references remote: Processed 1 references in total # Found a tag, generating the tarball remote: New tag pushed (vXX.YY.ZZ), generating the tarball... # Generating the tarball for Koha remote: [INFO] Archiving Koha git repo... # Retrieving the .po files from koha-l10n remote: [INFO] Cloning koha-l10n... remote: Cloning into '/tmp/po-XX.YY'... remote: Updating files: 100% (925/925), done. # Adding the .po files to the tar archive remote: [INFO] Appending .po files to the tar archive... remote: tar: Removing leading `/' from member names remote: tar: Removing leading `/' from hard link targets # Creating the .tar.gz file from .tar remote: [INFO] Compressing... # Copying the .tar.gz, tar.gz.MD5, tar.gz.MD5.asc and tar.gz.sig files to download.koha-community.org # Moving the old files to old_releases # Creating the koha-XX.YY-latest.tar.gz link remote: [INFO] Uploading the archive... # Generating the badge to display on the dashboard # And Copying to download.koha-community.org (https://download.koha-community.org/badges/git-tarball-XX.YY.svg) remote: [INFO] Generating the badge... remote: Connecting to img.shields.io (172.64.107.8:443) remote: saving to '/tmp/git-tarball-XX.YY.svg' remote: git-tarball-XX.YY.sv 100% |********************************| 974 0:00:00 ETA remote: '/tmp/git-tarball-XX.YY.svg' saved To git.koha-community.org:Koha-community/test.git + 7df25c1...4072ba4 vXX.YY.ZZ -> vXX.YY.ZZ
Email the package manager, after release is tagged
Email the package manager at 'mtj at kohaaloha.com' with subject 'Koha release XX.YY.ZZ', once you have pushed the new release tag.
Roll the release [skip]
Since November 2023 releases this is not longer needed. The tarball has been generating when you pushed the tag.
Public announcement
People should know there's a new Koha release. We announce it by email and by posting the release notes on the Koha Community site.
Post on the community site
The current community site is a Wordpress site. You should have a valid credentials (https://koha-community.org/wp-admin) and create a new Post. It should be titled 'Koha YY.MM.XX released'. We recommend using the Text editor instead of the Visual one. You can copy the HTML source code of the generated release notes into the Text editor with very few changes necessary:
Just change the first paragraphs of the generated release notes a bit, for example:
The Koha community is proud to announce the release of version 22.11.03. This is a maintenance release and contains many bugfixes and enhancements. As always you can download the release from: https://download.koha-community.org This is my first release as the 22.11.XX release maintainer. Thank you very much to everyone involved in this release. Please continue reading for the details this release. <!--more-->
Don't forget More entry (<!--more-->), that will split the announcement so it looks fine in the front page.
Choose the right categories: Announcements, release and <your branch>.
Email announcement
Email announcements are sent to the user list koha@lists.katipo.co.nz with a subject that should read like 'Koha 22.11.03 released'.
Note: keep in mind that the release announcement will be more than 40k and will be subject to moderation. So, be patient :-D
And that's it :-D.
Coordinated security releases
Sometimes it will be necessary to do a coordinated release of all maintained versions at the same time because of security related fixes. It can be a security-only urgent release or a non-urgent release following the usual schedule. The non-security workflow must still be followed but with some additions and changes:
- RM initiates a security release by emailing the RMaints and the Packaging Manager and provide them with the necessary information, like a list of bugs to push.
- RM pushes to the security/main branch in the protected security repository and informs RMaints.
- RMaints push the security patches from the security/main branch to the protected security repository. The non-security patches are still pushed to the normal branches.
- Important: As Jenkins won't run on the changes in the security branches, it's important to run the full test suite locally to avoid any unnoticed failures!
git remote add security git@git.koha-community.org:Koha-community/security.git git push security <branch>
- On the day of the release:
- In case of a non-urgent release, the normal branch must be merged into the security branch.
- For an urgent release, the security branch must contain only the security patches.The regular patches might have string changes which will be untranslated. So to avoid that, the security branch must be created from the last release, not from the public branch that might have additional patches. String changes in security releases can't be translated as the current workflow requires the patches to be in the public branch. And some time (a week) for the translators to have a chance to translate.
- RMaints generate the release notes as usual. The Bugzilla credentials in
release-tools/etc/user.yamlallow the release tools to access the security bugs.- Urgent release: fix the bug count wording in the release notes: https://gitlab.com/koha-community/release-tools/-/issues/30
- RMaints push the result (after applying the regular workflow) to the security branch and the tag to the security repository:
$ git push security v22.11.03
- RMaints create drafts of the release notes on koha-community.org - unpublished, but ready for publishing.
- Once an RMaint has finished those steps, they will inform the others.
- The Packaging Manager will build the packages from the security repository.
- The RMaint of the most recent/stable release (RMaint1) coordinates the last steps with the Packaging Manager.
- Non urgent release: RMaints push the content of the security branch to the public branch. No force push should be required as a merge was done earlier.
- Urgent release: RMaints merge the security branch into the public branch. And then push to the public branch. That way, no force push needed and the usual workflow can continue and the security patches are in the public branch.
- RMaints also push the tag to the public repository.
- RMaints publish the drafts on the website.
- RMaint1 and sends an email announcing the security release to the mailing list.
- The RMaint1 changes the bug status from security project to Koha project after pushing to main.
- RMaints change the bug status and add comments after that.
- Urgent release: RMaints go through the already backported but unreleased bugs and bump the "Version(s) released in:". Because the out of schedule security release caused the version of the next regular release to not be the same as planned.
- Tarballs will be generated as soon as the tag will be pushed to the public repository. So push to the public repository should be done soon after release.
- Urgent release: RMaints go through their db_revs directory to increment the version (the 3rd part of the number) of the pending/unreleased DBRevs. Those were on the public branch and not included in the security release. So their planned version has been replaced by the security release. Which mean that installs that upgraded to it won't have the DBRevs executed for those pending patches when they will be released next release.
- Urgent release: next regular release will be impacted: Release notes generation will have some things messed up. Namely the security patches will be listed again. Because to get the back ported patches before the urgent release. We will set the beginning of the commit range from February's release. So it will get the security patches again. Fix: manually remove them from the release notes. Along with the security fix section and fix the overall number of patches/fixes.
Backporting a patch with a new perl dependency
In manual or automated tests you will get an error about the missing dependency.
- Install manually the missing dependency and retest
- Ensure that the patch set adds the dependency to ./cpanfile (PerlDependencies.pm for 19.11.X)
- If not, do a followup patch if you can or ask for help
- Push
- Disregard the CI that will fail
- Wait for the koha-common package for your branch to be rebuilt, (rebuild happens every 60 mins)
- When it's done, wait for koha-testing-docker images to be rebuilt
- Rerun the CI, if you can't launch manual builds ask someone who have the rights
Tips and Tools
- One can use a slightly adapted version of the release manager pre-push hook to check your work before pushing to your maintenance branch: hook
- I use the follow git post-commit hook as a simple way to track whether a fix I've back ported has introduced a new author that should be added to the history.txt file and about page.
#!/bin/bash
AUTHOR="$(git log -1 --pretty=format:'%an' HEAD)"
FILE="/home/martin/kohaclone/docs/history.txt"
if grep -q "${AUTHOR}" ${FILE}; then
tput setaf 2; echo "Existing Author: \"${AUTHOR}\""; tput sgr0;
exit;
else
tput setaf 1; echo "New Author: \"${AUTHOR}\""; tput sgr0;
fi;
exit;
PGP/GPG and checksums clarifications
koha-$VERSION.tar.gz.sig
It allows to verify that koha-$VERSION.tar.gz has been signed by the RMaint's public key. Which needs to be added to the GPG keyring from a trusted source.
gpg --verify koha-$VERSION.tar.gz.sig ↓↓↓ Good signature from $NAME_OF_RMAINT
koha-$VERSION.tar.gz.MD5
It allows to check the integrity of koha-$VERSION.tar.gz against the checksum in koha-$VERSION.tar.gz.MD5 It doesn't guaranty that the file come from the RMaint and has not been compromised. It's just for checking that the download hasn't corrupted the file.
md5sum -c koha-$VERSION.tar.gz.MD5
koha-$VERSION.tar.gz.MD5.asc
It contains the same checksum as above but it's wrapped in a section and is followed by it's signature. Example:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 36002d5acf2a1ca5957d5f03cdfaac2d koha-20.11.07.tar.gz -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQRT86JTIbcYFBCjtfABQKyjt9hwuwUCYNrpxAAKCRABQKyjt9hw u8t/APwLhxg5Zh7qbOC8Dgko2+CxTepCPq4daThHzY80cEJjBwEA7v3xW9WOT5Cc tgRa9lpkUtin0baSZa7URcmgcxUceAQ= =Kxx0 -----END PGP SIGNATURE-----
Verify:
gpg --verify koha-$VERSION.tar.gz.MD5.asc ↓↓↓ Good signature from $NAME_OF_RMAINT gpg: WARNING: not a detached signature; file 'koha-$VERSION.tar.gz.MD5' was NOT verified!
The warning tells us that the .MD5 file isn't verified. Indeed, because only it's copy in the .MD5.asc file has been. So you would need to check that the .MD5 file content matches the one from the content section of the .MD5.asc file.
Release notes generation using the dedicated docker image
Get/Update the image
docker pull registry.gitlab.com/koha-community/release-tools
Run it to get a shell inside
docker run \
-v /PATH_TO_YOUR/release-tools/etc/user.yaml:/release-tools/etc/user.yaml \
-v /PATH_TO_YOUR/koha-code:/koha \
-v /OPTIONAL/PATH_TO_YOUR/persistent-files/bash_history-release-tools-root:/root/.bash_history \
-ti registry.gitlab.com/koha-community/release-tools \
bash
Generate the release notes
Example with releasing v20.05.15
export PERL5LIB=/koha
cd release-tools/
git pull https://gitlab.com/koha-community/release-tools main
cd /koha/
../release-tools/bin/koha-release v20.05.14..HEAD notes
chown 1000:100 misc/release_notes/*
# see normal workflow for the details
# fix the release notes belonging to root. UID and GID might need to be something else than 1000 and 100 to match your host system
../release-tools/bin/koha-release v20.05.14..HEAD html misc/release_notes/release_notes_20_05_15.html
chown 1000:100 misc/release_notes/*