Benchmark for 16.11
To follow the previous Benchmark for 3.22, I have decided to replay the same script on different versions of Koha to compare them. Now that plack and memcached are considered as stable, the idea is to highlight the fact that using them should bring a lot of performance improvements in new versions.
Conditions
The script
The same selenium script has been used.
The goal is to benchmark different actions a Koha user is doing regularly.
The script can be found in the Koha codebase at t/db_dependent/selenium/basic_workflow.t
It collects the execution time spent to execute different kind of actions:
- main => hit mainpage.pl and login
- add patron category => hit admin/categories.pl, fill the form and submit it
- add patron => hit /members/memberentry.pl?op=add&categorycode=test_cat, fill the form and submit it
- add biblio => Add 3 bibliographic records without using the interface, we will forget this step
- add item => Create 3 items (1 per bibliographic record) by hitting /cataloguing/additem.pl?biblionumber=X We don't fill the form (too DB dependent), just an empty form
- checkout => Check the 3 items out to a patron: hit /circ/circulation.pl?borrowernumber=X, fill the barcode form and submit it
- checkin => Return the 3 items: hit /circ/returns.pl, fill the barcode form and submit it
- search => Search for "s", then "d" (/catalogue/search.pl?q=s, /catalogue/search.pl?q=d). Both search will return 20 results
The last (search) step is a new one I have just added for this benchmark.
Environment
I have used a kohadevbox on a i5-6200U with 8Gio of RAM (2Gio are allocated to the box), SSD
Setup
I have written a new script benchmark_them_all.pl in the koha-misc4dev repository.
The Koha versions 3.14, 3.16, 3.18, 3.20, 3.22, 16.05 and 16.11 have been compared. To do so a new branch perf_${version}.x has been pushed to my Koha github repository
See the README file (section "You want to benchmark Koha") to know how to replay the benchmarks by yourself.
Please replay them to get more numbers and confirm (or not) mine.
Important: The branches used are the latest ones, so much more different from the original releases (3.18.0 vs 3.18.13)
The data
| x | 3.14 | 3.16 | 3.18 | 3.20 | 3.22 | 3.22 Plack | 16.05 | 16.05 Plack | 16.11 | 16.11 Plack |
|---|---|---|---|---|---|---|---|---|---|---|
| main | 2.316 | 2.34 | 2.422 | 3.982 | 4.156 | 1.174 | 4.4 | 1.1 | 4.284 | 1.102 |
| add patron category | 3.164 | 3.236 | 3.404 | 3.95 | 6.738 | 2.078 | 7.198 | 2.266 | 7.27 | 2.32 |
| add patron | 4.374 | 5.206 | 6.18 | 5.642 | 6.726 | 1.89 | 6.882 | 1.94 | 6.944 | 2.096 |
| add items | 10.988 | 11.008 | 11.7 | 12.236 | 15.15 | 5.224 | 15.868 | 5.164 | 16.658 | 6.636 |
| checkout | 7.716 | 14.8 | 12.986 | 14.266 | 14.998 | 5.06 | 16.038 | 5.23 | 15.388 | 5.838 |
| checkin | 6.334 | 7.052 | 6.828 | 7.348 | 12.884 | 3.478 | 14.498 | 4.06 | 14.4 | 4.226 |
| search | 3.338 | 3.434 | 5.798 | 6.218 | 7.656 | 5.854 | 7.17 | 4.32 | 6.638 | 3.928 |
And the handsome graph
Quick conclusion/interpretation
- Compare with the previous benchmarks, we do not find back the improvements brought by 3.20. I have tried several times and always get the same graph. It might come from wrong data generated previously (or things got pushed between 3.20.0 and 3.20.15).
- There is still the gap between 3.20 and 3.22 for the "add items" step. And it is still increasing between 16.05 and 16.11. It seems that we have something to have a look and improve in this area.
- The execution time for the search step is decreasing since 3.22 (certainly comes from the marc structure data we cache, see 16431 and 16365)
Let's go further
It's time to compare what bring us memcached and plack.
I'd like to know if memcached bring us lot of performance improvements.
I have slightly modified the benchmark script (see branch plack_vs_memcached) to execute the same scripts given different factors:
- memcached and plack are stopped (16.11)
- memcached is started but plack is disabled (16.11_memcached)
- memcached is stopped but plack is enabled (16.11_plack)
- memcached and plack are started (16.11_plack_and_memcached)
- I created a new commit on top of the perf_16.11.x branch wich makes Koha::Cache->set_in_cache does nothing, i.e. returns directly (16.11_plack_no_cache)
We can see that plack brings us a lot compared to memcached.
It's surprising not to see a much bigger difference between 16.11_plack and 16.11_plack_and_memcached (there is something wrong here!)
But 16.11_plack_no_cache shows us that our caching system is very useful (at least the L1 cache) for the search and add items steps.
I have to admit that I would have expected 16.11.x be better than 3.22.x
What's next?
Bug 17819 - Performance 16.11 - Omnibus will collect the different performance improvements found for 16.11
Template caching
Bug 7533 - Add template_cache_dir to the koha-conf.xml file will add the template_cache_dir to the koha-conf.xml file.
It brings a huge performance boost.
