Release maintenance

From Koha Wiki

Jump to: navigation, search
Home
Home
Koha > Technical > Development
Koha > Technical > Development > Release Management

Note: this guidelines are work in progress, and release maintainers have the right to change this to whatever they prefer. For now this will serve as a documentation place for the tasks involved in release maintenance.


Contents

The Release Maintainer role

Release maintainers do the task of keeping the stable Koha releases in line with patches provided for bugfixes. People are elected for this task after they have volunteered for it, and there is consensus on the development community for them to take on the job.

RMaints are assigned one or more stable branches to maintain. Crazy people like Chris Cormack, Chris Nighswonger and Fridolin Somers have maintained more than one release at a time. It's a crazy Chris thing®.

Schedule

Every month there is a stable maintainance release. The relevant dates are:

  • 15th: String Freeze
  • 22nd: Release

String Freeze: During String Freeze patches that change strings should not be pushed! Please keep in mind that a changed string will mean that the string appears untranslated after updating, which can be especially toublesome for OPAC strings. Most users rely on the packages to provide translations and are not aware how they could fix such problems locally.

  • Get in touch with the translation manager on time to have them update the po files on translate.koha-community.org
  • Announce String Freeze on the koha-translate mailing list

Release: After each release, get in touch with your fellow release maintainers and the packaging manager in order to agree about a date for string freeze, a date for when your branches will all be ready for packaging and a date for the release. This is thought to make the life of our packaging manager easier as they can build all branches at once. Also it means that the packages will be ready at release.

Initial setup

Request push access for git

You will need to be given privileged access to the community git server.

To enable this, you will need to send a copy of your public ssh key to Chris Cormack, Galen Charlton or Martin Renvoize.

Create a remote for pushing

You could set up a remote for pushing:

  git remote add upstream gitmaster@git.koha-community.org:koha.git

and then push using this:

  git push upstream 18.11.x

Git pre-push hook

There is a pre-push hook documented in the 'Tips & Tricks' page, this can be beneficial as it adds a series of final checks before allowing you to push your local updates to the public repository.

Release tools

This section lists the set of tools a release maintainer needs to be able to make releases.

The release-tools repo

The Koha dev community has a nice set of tools to make the job easier. This set of scripts is on a git repo that can be cloned using

$ git clone https://gitlab.com/koha-community/release-tools.git

GPG keys

In order to sign the git version tag you need to have valid gpg signatures. Generate them issuing:

$ gpg --gen-key

Keys can be generated using 4096 or 2048 bits. It took me a while and lots of USB HDD activity to get enough entropy but I managed to generate it with 4096 bits.

You may need to configure your git repo, see here.

You can add the key quickly to a repo (or all repos) with

$ git config --global user.signingkey [gpg-key-id]

Handover between two cycles

The old and new release maintainer of a given branch should contact each other to share this information:

  • The position of the current "RMaint cursor" i.e. the last bug considered for backporting.
  • The eventual list of bugs that due to the last string freeze have not been yet backported.
  • Any other significant relevant info like gotchas of the branch (in conflicts, in random automated tests failures, etc).

Daily maintenance workflow

Release Maintainers usually watch the activity on the master git branch. If they maintain a release other than the latest stable release, they should be looking at higher-numbered versions git logs (for example, if you are maintaining the 3.10.x stable branch, then you should be probably watching 3.12.x's, and keep in touch with its maintainer).

MRenvoize uses a quick and dirty script to add bugs from master into hist todoist list.

You should also watch Bug 17108 - Automatic debian/control updates for patches specific to your version as they may well not hit master ;)

Once you spot a specific commit or commit series on your queue, the first thing is to look at bugzilla and check if it is a bugfix or an enhancement and decide whether to back port.

Deciding on what to back port

The general principle is to maintain stability and don't break things for supported releases!

Release Aim for the release Guidance for back porting
all releases Stable and reliable
  • Bug fixes: all critical and blocker bugs
  • Security fixes
  • API and plugin enhancements: simple, non-breaking enhancements only
stable Most feature-rich release, there may be some bugs
  • Other bug fixes: try and backport regardless of severity, very minor bug fixes may be skipped
  • Enhancements: case-by-case, generally only minor enhancements if they don't have database changes, no new features
oldstable Tried and tested, more bug-free
  • Other bug fixes: try and backport regardless of severity, very minor bug fixes may be skipped
  • Enhancements: case-by-case, generally only minor enhancements if they don't have database changes, no new features
oldoldstable The most bug-free release
  • Bug fixes with a major severity: try and backport, but not if it may be risky or it is later in the cycle and release will soon become end of life
  • Bug fixes with a normal severity and below: don't backport and leave a message that it can be done on demand
  • No enhancements or new features

Other notes:

  • Enhancements: Generally only backport minor enhancements, leave a message on the bug to say it can be done on demand if required.
  • Seek clarification where required: Check with the rest of the devs on IRC or mailing the author of the patches for clarification.
  • Changes with database updates: Be particularly cautious about backporting a change that includes a database update. All database updates 'should' be idempotent, meaning they can be run repeatedly without negative consequences. It's always worth double checking this as errors have been known to creep through occasionally. For example, a DB update that adds a new table will result in a warning, at least, during upgrade, and should be avoided unless it is absolutely required in order to resolve a security bug. When it doubt, please discuss with the RM and fellow Release Maintainers first.

When not back porting

Mark the bug as "Resolved - Fixed."

Processing a commit or commit series

Once you have decided the patch should be backported to the stable branch you are maintaining, update your local git repository (just in case) and create a working branch. You should define a naming schema to avoid confusion now you have commit rights! As 18.11.x release maintainer I chose 18.11bug_<bugnumber>. Do what you want :-D

  git checkout upstream/18.11.x -b 18.11bug_<bugnumber>

You should then look for the commit hash of the patch(es). And then cherry-pick them like this:

git cherry-pick -x 1a8db0ba7d62a70cd6b49ee705a5db9b11ad7672

If there's some conflict with cherry-picking, git will tell you. You will have two options

  • Solve the conflict
  • Try to solve the conflict and give up

If solving the conflict is not straightforward, comment on the bug like this: 'This does not apply cleanly to 18.11.x please submit one formatted for 18.11.x'.

If you solved the conflict you finish by

git add <touched files>
git commit

A cherry-pick message is added to the commit, and if there were conflicts, on which files.

Note: If the patchset includes any database updates, you will need to correct the version number bump to match your branch and running number.

Note: If the patchset includes changes to database table structures you will likely need to rebuild the DBIx::Class schema files using the included /misc/devel/update_dbix_class_files.pl script

Note: If the patchset includes CSS changes, you will likely need to rebuild the CSS files from their source SCSS Working_with_SCSS_in_the_OPAC_and_staff_client

You should run some tests. For example, if the patch modifies several templates, you should at least run:

perl xt/tt_valid.t
perl xt/author/valid-templates.t

Do some testing, and keep in mind you should be confident about the QA team.

Once everything is solved, and testing proves everything is fine with the cherry-picked commits, you should sign on them:

git commit --amend -s

And then merge the changes with your maintenance branch

git checkout 18.11.x
git merge 18.11bug_<bugnumber> --ff-only

Pushing upstream

Once you've collected some bugs in your local branch, you need to remember to push it up to the community.

 git push upstream 18.11.x

Assuming you have configured the git pre-push hook, using this branch should run a final pre-push check on your work before allowing you to push your changes up to the wider world.

Update Bugzilla

This is very important: Let people know you pushed the patches to your stable branch by changing the status to Pushed to stable/oldstable/oldoldstable and we also now have a custom field (Version(s) released in:) in Bugzilla which you should add the specific release version to for which the bug will be released (This field takes the form of a comma-delimited list of versions, for example, `19.05.00, 18.11.03, 18.05.11`).

Note: You could use the `koha-push` script from the release-tools repository to do the bugzilla update. It will do a final check on the bug status, update the status, add text to 'Version(s) released in' and add a comment to the bug for you.

Monitoring Jenkins

We use Jenkins, a CI tool, to test each branch after new code is pushed to it. You should have been added to the email contacts for failing jobs and as such should get notified of any test breakages shortly after pushing any changes.

You can view the progress of the build tests on the Koha community Jenkins site. For the 18.11 branch, this takes about 30 minutes for each build.

Releasing

Push updated translation files

Keep in touch with the translation manager to get the strings on translate.koha-community.org updated in time for String Freeze on the 15th of a month.

Right before release, you can use a form provided by the translation manager, to automatically create a new branch on the TM's git repository with the latest version of the po files. Note: As the form is not protected at the moment, you can get the link to it from one of the previous Release Maintainers. Also, when inputting the version number, the format is 19.05.12, not v19.05.12.

You will receive an email with all necessary information about the created branch and can then merge it into yours:

$ git fetch remotename
$ git merge remotename/branch

Remember to sign the relevant patches.

You must check the po files for errors, you can do:

sudo apt-get install translate-toolkit
junitmsgfmt misc/translator/po/*
# search for 'fatal error' in the output

An error can be fixed, for example, by marking the string as 'fuzzy' in the po file and re-running the check.

Errors (not warnings) can lead to major translation breakage not just on the invalid string. e.g. only half the files get copied into the translated directories. Or a broken page due to a missing variable causing program code to end up in the page.

You will also need to check for possible XSS attempts:

 The POs must *not* contain:
 % git grep '<script' **/*.po
 % git grep PERL **/*.po # This has 4 occurrences, but not valid

You should then commit those changes:

$ git add misc/translator/po
$ git commit -m 'Fix translations for Koha YY.MM.XX'
$ git commit --amend -s

Bump version number

In Koha.pm increment the following line:

 $VERSION = "YY.MM.XX.000";

Remember that between releases there might be minor database version upgrades. They would have been done the same way, using the latest minor number to reflect those version changes. For example 17.05.02.001, 17.05.02.002 and so on.

Add this database upgrade to installer/data/mysql/updatedatabase.pl:

If Koha >= 20.05 :

$DBversion = 'YY.MM.XX.000';
if( CheckVersion( $DBversion ) ) {
    NewVersion( $DBversion, "", "Koha YY.MM.XX release" );
} 

Else :

$DBversion = "YY.MM.XX.000";
if ( CheckVersion($DBversion) ) {
    print "Upgrade to $DBversion done (YY.MM.XX release)\n";
    SetVersion ($DBversion);
}

You should then commit those changes:

$ git add Koha.pm installer/data/mysql/updatedatabase.pl
$ git commit -m 'Increment version for YY.MM.XX release'
$ git commit --amend -s

You can now push the latest changes. Or continue with the next step.

Generate the release notes

NOTE: Make sure to pull the latest version of the release tools before generating the notes

Release notes are automatically generated by the koha-release script from the release-tools repo. If you are releasing version 17.05.03 you should run this on your clean 17.05.x branch:

$ ~/<path_to>/release-tools/bin/koha-release v17.05.02..HEAD notes
# The HTML version is generated from the Markdown one.
# So if editions need to be made, they can be done to the Markdown version only and the HTML can be regenerated.
$ ~/<path_to>/release-tools/bin/koha-release v17.05.02..HEAD html misc/release_notes/release_notes_17_05_03.html

it will create the following files:

- misc/release_notes/release_notes_17_05_03.md
- misc/release_notes/release_notes_17_05_03.html

System requirements and recommendations

Replace the content of the section "System requirements" with a link to this page. It can look like that:

You can learn about the system components (like OS and database) for running Koha here: https://wiki.koha-community.org/wiki/System_requirements_and_recommendations

Check for inconsistency

It is important that you edit that file and fix any inconsistency you find. Usually

- If there are no features/enhancements/bugfixes the current script may generates empty sections for them.
- The branch name might be wrong at the end of the file.
    It will likely be wrong when doing a security release (e.g. fix it with 20.05.x instead of 20.05.x-security)
- The team might be wrong if the config file has the wrong master version. (etc/user.yaml)
- The system requirements and recommendations should point to the wiki page.

Just make sure everything is sound. Once you are done, you should commit the release notes:

$ git add misc/release_notes
$ git commit -m 'Update release notes for YY.MM.XX release'
$ git commit --amend -s

It's not a bad idea at this point to push your branch upstream one last time and watching all the tests pass before tagging it for release.

Tag the release

Tagging will prompt for your GPG passphrase. It is done issuing:

$ git tag -a v17.05.03 -s -m 'Koha release 17.05.03'

Note the two digits. You can now push the new tag doing:

$ git push upstream v17.05.03

Roll the release

This step creates the release tarball. It needs the release-tools repo. You have to run

$ mkdir releases
$ ~/path-to/release-tools/roll_release.sh 17.05.03 local_release_branch

↑↑↑ watch out for the local_release_branch in case of a security release when you might use another local branch than usually.

It will create the following files under the releases dir:

  • koha-17.05.03.tar.gz
  • koha-17.05.03.tar.gz.MD5
  • koha-17.05.03.tar.gz.MD5.asc
  • koha-17.05.03.tar.gz.sig

You then have to upload the generated files for distribution:

$ scp releases/koha-17.05.03.tar.gz* download.koha-community.org:/home/uploads/files/

Then log into the server.

Older versions of the same release should be moved into the old_releases directory.

If you create latest stable release, recreate symlink koha-latest.tar.gz.

Public announcement

People should know there's a new Koha release. We announce it by email and posting it on the Koha Community site.

Post on the community site

The current community site is a Wordpress site. You should have a valid credentials and create a new Post. It should be titled 'Koha YY.MM.XX released'. We recommend using the Text editor instead of the Visual one. You can copy the HTML source code of the generated release notes into the Text editor with very few changes necessary:

Just change the first paragraphs of the generated release notes a bit, for example:

  The Koha community is proud to announce the release of version 17.05.03.
  This is a maintenance release and contains many bugfixes and
  enhancements.
  
  As always you can download the release from:
  
  https://download.koha-community.org
  
  This is my first release as the 17.05.XX release maintainer.
  Thank you very much to everyone involved in this release.

  Please continue reading for the details this release.

  <!--more-->

Don't forget More entry (<!--more-->), that will split the announcement so it looks fine in the front page.

Choose the right categories: Announcements, <your branch> and release.

Email announcement

Email announcements are sent to the user list koha@lists.katipo.co.nz with a subject that should read like 'Koha 17.05.03 released'.

Note: keep in mind that the release announcement will be more than 40k and will be subject to moderation. So, be patient :-D

And that's it :-D.

Coordinated security releases

Sometimes it will be necessary to do a coordinated release of all maintained versions at the same time because of security related fixes. I can be a security-only urgent release or a non-urgent release following the usual dates. The non-security workflow must still be followed but with some additions and changes:

  • RM initiates a security release by emailing the RMaints and the Packaging Manager and provide them with the necessary information, like a list of bugs to push.
  • RMaints push the security patches from Bugzilla to the protected security repository. The non-security patches are still pushed to the normal branches.
git remote add security gitmaster@git.koha-community.org:security.git
git push security HEAD:new/security-release-foo
  • On the day of the release, in case of a non-urgent release, the normal branch must be merged into the security branch.
  • RMaints generate the release notes as usual. The Bugzilla credentials in release-tools/etc/user.yaml allow the release tools to access the security bugs.
  • RMaints push the result (after applying the regular workflow) to the security branch and the tag to the security repository:
$ git push security v17.05.03
  • RMaints roll tarballs and upload them to their non-public home directory on the download server:
$ scp releases/koha-17.05.03.tar.gz* download.koha-community.org:/home/<youruser>/
  • RMaints create drafts of the release notes on koha-community.org - unpublished, but ready for publishing.
  • Once an RMaint has finished those steps, they will inform the others.
  • The Packaging Manager will build the packages from the security repository.
  • The RMaint of the most recent release (RMaint1) coordinates the last steps with the Packaging Manager. RMaint1 then moves the tarballs into the public directory.
  • RMaints push the content of the security branch to the public branch. No force push should be required as a merge was done earlier.
  • They also push the tag to the public repository.
  • RMaints publish the drafts on the website.
  • RMaint1 and sends an email announcing the security release to the mailing list.
  • The RMaint1 changes the bug status from security project to Koha project after pushing to master.
  • RMaints change the bug status and add comments after that.

Backporting a patch with a new perl dependency

In manual or automated tests you will get an error about the missing dependency.

  1. Install manually the missing dependency and retest
  2. Ensure that the patchset adds the dependency in PerlDependencies.pm
    1. If not, do a followup patch if you can or ask for help
  3. Push
  4. Disregard the CI that will fail
  5. Wait for the koha-common package for your branch to be rebuilt
    1. Monitor this: http://debian.koha-community.org/koha-staging/pool/main/k/koha/?C=M;O=D
  6. When it's done, wait for koha-testing-docker images to be rebuilt
    1. Monitor this: https://gitlab.com/koha-community/koha-testing-docker/-/pipelines
  7. Rerun the CI, if you can't launch manual builds ask someone who have

the rights


Tips and Tools

  • One can use a slightly adapted version of the release manager pre-push hook to check your work before pushing to your maintenance branch: hook
  • I use the follow git post-commit hook as a simple way to track whether a fix I've backported has introduced a new author that should be added to the history.txt file and about page.
#!/bin/bash

AUTHOR="$(git log -1 --pretty=format:'%an' HEAD)"
FILE="/home/martin/kohaclone/docs/history.txt"

if grep -q "${AUTHOR}" ${FILE}; then
 tput setaf 2; echo "Existing Author: \"${AUTHOR}\""; tput sgr0;
 exit;
else
 tput setaf 1; echo "New Author: \"${AUTHOR}\""; tput sgr0;
fi;

exit;
Personal tools