Sandboxes

From Koha Wiki
Jump to navigation Jump to search

Sandboxes

Sandboxes have been developed with the goal to lower the barrier for testing patches for Koha.

You can set up a sandbox with a current development version of Koha, sample data and the patches of a bug applied for you. Once you have finished testing, you can use a form to 'sign off' on the patches and move the bug report to the 'Signed off' status from where it will continue into QA.

The code of the sandboxes and documentation on how to set them up has been released under GPL and can be found in the Koha Community gitlab account.

Available sandboxes

BibLibre

BibLibre Sandboxes

login/password koha/koha

If you encounter a problem with these sandboxes, please send an email to sandboxes at biblibre dot com or ask on the IRC channel.

ByWater Solutions

ByWater Solutions Sandboxes

login/password koha/koha

If you encounter a problem with these sandboxes, please contact khall on the IRC channel.

Open Fifth

Open Fifth Sandboxes

login/password koha/koha

If you encounter a problem with these sandboxes, please send a mail to sandboxes at ptfs-europe dot com or contact Martin Renvoize (aka ashimema) on Mattermost.

Documentation and Howtos

There are some great tutorial videos that explain the whole process:

Step by step

Step 1: Set up a sandbox

It takes some time to startup so we do it first.

  • Pick one of the sandbox providers above
  • Click on the Create link on top of the page
  • You will see a form with many options, but for a minimal setup, you only need to fill in a few of those marked with *.
    • Your name*: If you end up signing off on a bug, this will be used as the name in your sign-off line.
    • Your email*: This is the e-mail address that will be used in your sign-off line, when you decide to sign off on a bug after testing.
    • Sandbox name*: Name your sandbox using lower case letters and numbers only.
    • Bug number: Don't fill it. A lot of test plans start with testing Koha without any patch to reproduce a bug for example. So a lot of the times applying a patch will be done in the middle the test instructions. Plus, if the patch fails to apply at this step, it won't be clearly shown. Whereas if applied later with the dedicated tool, the error will be shown.
    • Marc flavour: Possible options here are MARC21, UNIMARC. The MARC flavour you pick will affect the configuration, sample data and index configuration.
    • Git remote & branch: Only used rarely for a few patches. This allows to test bugs where the patches are not attached to Bugzilla for their size, but are on a remote branch of the developer. It also allows to set up a sandbox with a very specific Koha version. (not too old, else the Perl dependencies won't match) If left empty the latest development version plus the patches from Bug number will be used.
    • Description: optional field to make some note about what this sandbox will be used for.
    • Safety question*: What was the world's first open source ILS? - We assume you know the correct answer, otherwise refer to the hint below the field :)
  • Click Submit
  • The Status column will first read Provision pending while the installation with your selected options is built on the server. Refresh the page after a bit and when it reads Provisioned your sandbox is ready to go. Meanwhile you can go on with the next step.

Step 2: Select a bug to test in Bugzilla

Bugs in Bugzilla can be bug fixes, small enhancements or complete new features. As there is always a lot going on, selecting a bug can take some time.

You don't need an account to search in Bugzilla, but in order to make status changes and to comment, you will have to register with an e-mail address.

An easy way to see a list of all bugs waiting for sign-off is using the link from the Koha Dashboard(section "Bug statuses => need signoff").

↓↓↓Here they are directly↓↓↓:

You can also create a list of all bugs in the 'Needs Signoff' status by using Bugzilla's Advanced search. Just click on 'Needs Signoff' in the status field and start the search.

The component column will give you an idea about the area in Koha this bug applies to. "Templates" bugs are often a good place to start with sandbox testing.

Finding the up to date test plan in a bug

Attachments section => click the date of the 1st listed attachement (of the type "patch") to make the page scroll down to it. This should be the patch containing the test plan. If not, look at the other attachements after it. It looks like some variation of this: 

Test plan:
1. Step to prepare some stuff
2. Another step
3. Do something else that exposes the current issue
4. Apply the patch
5. Redo some of the above steps
6. See that the issue is gone

Before these latest attachements, there would have been a lot of other patches attached. These are the previous, now obsolete versions. (with likely different test plans) The attachments section of a bug page only shows the latest.

What is testable or not in a sandbox

There are some limits to sandbox testing, to make it easier, we have created 2 keywords that can be set:

  • no-sandbox: this bug is for one reason or another not testable on a sandbox.
  • Sandbox: this bug is testable on a sandbox!

The keywords are not always used (yet), so the follow applies to bugs without keywords:

Quickly decide, but risk missing actually testable tickets

  • Test plan mentions running a command line script:
    • If it's a unit test file ending in .t or it reads "prove ..." you can skip this step and still test, but please note it on the bug report. QA team will run the tests later.
    • If it's any other type of script, probably ending in .pl, you won't be able to test, so skip this bug. If you can, set the 'no-sandbox' keyword.

Assess testability

Not testable in a sandbox:

  • Editing or copying files like koha-conf.xml, XSLT stylesheets, Zebra configuration like biblio-koha-indexdefs.xml, ccl.properties, authorities.xml
  • Run an UPDATE/DROP/INSERT SQL query. These can only be run from command line. Any SELECT statement can be run using reports.
  • Update translation strings. This is not the same as installing a translation.
  • Run CLI scrpits like overdue_notices.pl or longoverdue.pl
  • LDAP, Shibboleth, OAuth
  • SIP
  • Installing a new Perl library/dependency
  • very likely any command that you don't see matching what's in the next list

Testable in a sandbox:

  • Run tests (the "prove" command): we can't do it, but you can just skip this step and note it in the comment. The QA team will take care of it later.
  • Install translations, it's possible to deploy a few languages via sandbox management interface > Actions > Install translations. This is different from updating translation strings.
  • Restart services, restart Plack or Memcached. Sandbox management interface > Actions > Restart services
  • Rebuild the staff interface or OPAC CSS: Sandbox management interface > Actions > Build CSS
  • build-resources.PL same as above: Build CSS
  • Refresh schema, run dbic: Sandbox management interface > Actions menu > Refresh schema
  • Send emails, including process_message_queue.pl see Sandboxes/Testing sending emails
  • Run a SELECT SQL query: use SQL reports
  • Check log files: Sandbox management interface > Logs
  • Use Elasticsearch: you need to test this right away before starting doing the test plan. Between the 3 sandbox providers they don't all have a working Elasticsearch setup. So change system preference "searchengine" to Elasticsearch and go the sandbox management interface > Actions > Full Elastic Reindex. Then when it's done, try a catalogue search.
  • Install plugins: you need to test this right away before starting doing the test plan. Between the 3 sandbox providers they don't all allow plugin installation (probably for security reasons). So in your sandbox try to install the kitchen-sink plugin, if it doesn't fail and your interface turns orange, then it works. Then uninstall it and you can test the patches.


Complete this list with any missing case or report it to the koha-devel mailing list or the chat. So we can get this exhaustive enough to make patch testing the most autonomous possible.

Step 3: Login and testing

  • You can now use the buttons to Staff and OPAC interface of your created sandbox from the management page. Logins and passwords are listed for each provider above.
  • Start testing referring to the test plan (and beyond).
  • When the test plan instructs you apply the patch: Go to the management page => actions apply patches => enter the bug number (top left of a bug page)
    • If when submitting, the page displays ends up displaying a long confusing message, then it failed to apply the patch. So copy the message in a comment on the bug and change the bug status to "patch doesn't apply". Else it worked and you should see a short message that looks like it asks you to apply a patch. It's not actually a question, it has been automatically answered by the sandbox automatically.
  • Restart services after applying a patch. It's an implicit step of many test plans. When applying a patch, services needs to be restarted
  • If you run into problems in other steps, please add a description of it as a comment in Bugzilla and set the status to Failed QA. This can also be used if there is no test plan or if you are stuck on something to get the developer's attention.
  • If everything is working right and looking good, please add your sign-off!

Step 4: Sign-off

  • Return to the Sandbox management page. Select Sign-off patches from the Actions pull down on the right.
    • Your name: your name. It will be part of the sign-off line added to the patch files.
    • Your email: your e-mail address. This will also be part of the sign-off line. Together with your name this information will be used for building the release notes and for the Dashboard.
    • Bug number: the number of the bug that you applied the patches from earlier on creating the sandbox.
    • Number of patches to sign: this can be a little tricky. Often times it will be just 1, but there are bugs that can have a lot of patch files attached. In this case, count the attachments marked as patch in the table on top of the bug description in Bugzilla and enter the number here.
    • Anti-spam: Just enter Koha here.
  • Submit
  • Refresh to bug page on Bugzilla. You should see new comments, they are new attachement and you should see they have your sign-off line.
  • Important: At the moment signing off from the sandbox will attach the patch with your sign-off line, but it won't change the status to Signed off. Please do so manually in Bugzilla, so that it will be moved into the queue of the QA team!
  • Congratulations! Now you can delete your sandbox from the management page and start over creating a new one to test more patches :)

Tips and tricks

Testing sending emails

If you want to test a bug where you have to check an email being sent, you must specify a valid looking e-mail address in the system preference KohaAdminEmailAddress. Like KohaAdminEmailAddress@example.org

But because the sandboxes do not actually send out e-mails, you will need to check two places depending on the Koha feature:

Direct emails

  • sending a cart via email
  • serial issue claims and acquisitions claims for late orders
  • email order feature
  • serial alerts: when you subscribe to get an email of a new serial issue
  • complete this list if something is missing
    • what about: patron page => more => send welcome email

For these go to the sandbox management interface => logs => koha mail log

For the rest, jump to next section.

The koha mail log displays the message in a raw format that changes some characters and URLs. So you see weird stuff and are unsure, paste the content on of the email from the mail log to this decoder: https://www.motobit.com/util/quoted-printable-decoder.asp Paste the email and click decode. And then the first/top text area contains the result.

Emails put in queue for sending them periodically

Use an SQL report to view the content of the generated e-mails found in the message_queue table. Test plans usually tell to run process_message_queue.pl to have them sent but we can directly look at the queue.

select * from message_queue;

Rebuild CSS from SCSS

Sometimes you will see soomething like the following in a patch set: 

To test, apply the patch and rebuild the staff interface CSS
(https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client).

In this case, after spinning up the sandbox and applying the patch, go to Actions > Build CSS.


Tester handbook

Retrieved from "https://wiki.koha-community.org/w/index.php?title=Sandboxes&oldid=38662"