Benchmark for 3.22
Between Koha 3.16 and 3.18, users faced some performance issues.
To understand, the code has been benchmarked, see bug 13690, especially the comment 2.
The 3.22 release is getting close and it would be great to know what users can expect in term of performance.
To get these values, the selenium tests on bug 13691 have been used.
Raw data
v3.16.00
CP main = 3.37 + 3.28 + 3.40 + 3.40 + 3.37 = 3.36
CP add patron category = 4.12 + 4.23 + 4.03 + 3.75 + 4.09 = 4.04
CP add patron = 4.15 + 4.19 + 4.22 + 4.46 + 4.17 = 4.24
CP add biblio = 1.00 + 1.30 + 0.84 + 0.79 + 0.87 = 0.96
CP add items = 13.89 + 12.80 + 12.68 + 12.69 + 12.47 = 12.91
CP checkout = 11.92 + 11.39 + 11.45 + 11.09 + 11.34 = 11.44
CP checkin = 9.74 + 9.65 + 10.01 + 8.97 + 9.04 = 9.48
v3.18.00
CP main = 4.64 + 4.65 + 4.69 + 4.79 + 4.61 = 4.68
CP add patron category = 6.06 + 5.96 + 6.75 + 6.26 + 5.99 = 6.20
CP add patron = 6.76 + 6.45 + 6.59 + 6.65 + 6.48 = 6.59
CP add biblio = 1.06 + 1.06 + 1.13 + 1.09 + 1.11 = 1.09
CP add items = 18.93 + 17.03 + 17.50 + 17.26 + 17.41 = 17.63
CP checkout = 16.48 + 16.80 + 15.52 + 15.20 + 15.42 = 15.88
CP checkin = 16.53 + 13.92 + 13.61 + 13.85 + 14.10 = 14.40
v3.20.00
CP main = 4.63 + 4.91 + 4.83 + 4.89 + 5.05 = 4.86
CP add patron category = 4.49 + 4.56 + 4.58 + 4.45 + 4.39 = 4.49
CP add patron = 6.03 + 5.88 + 5.90 + 5.94 + 5.71 = 5.89
CP add biblio = 2.17 + 1.03 + 0.86 + 1.24 + 0.98 = 1.26
CP add items = 13.86 + 13.55 + 13.37 + 13.56 + 13.38 = 13.54
CP checkout = 16.03 + 15.86 + 16.06 + 15.91 + 16.08 = 15.99
CP checkin = 10.22 + 9.66 + 10.18 + 10.03 + 10.07 = 10.03
master (3.22 f25fe6dd Fri Now 6 11:21:56 2015)
CP main = 4.98 + 5.13 + 5.16 + 5.46 + 5.14 = 5.17
CP add patron category = 7.32 + 7.22 + 7.58 + 7.17 + 7.30 = 7.32
CP add patron = 7.39 + 7.13 + 7.36 + 7.38 + 7.11 = 7.27
CP add biblio = 1.41 + 1.14 + 0.91 + 0.94 + 1.11 = 1.10
CP add items = 19.63 + 20.15 + 20.61 + 19.94 + 20.30 = 20.13
CP checkout = 17.06 + 17.69 + 17.77 + 17.02 + 17.68 = 17.44
CP checkin = 16.13 + 15.90 + 14.95 + 15.29 + 15.74 = 15.60
v3.16.00 under Plack
CP main = 2.63 + 2.21 + 2.67 + 2.55 + 2.18 = 2.45
CP add patron category = 2.01 + 1.91 + 1.98 + 1.82 + 2.05 = 1.95
CP add patron = 2.19 + 2.33 + 2.21 + 2.40 + 2.22 = 2.27
CP add biblio = 1.03 + 0.99 + 0.79 + 0.89 + 1.03 = 0.95
CP add items = 9.52 + 9.91 + 9.02 + 9.26 + 9.08 = 9.36
CP checkout = 6.25 + 6.12 + 6.44 + 6.13 + 6.08 = 6.20
CP checkin = 5.96 + 5.81 + 5.83 + 5.03 + 5.24 = 5.57
v3.18.00 under Plack
CP main = 2.66 + 2.40 + 2.45 + 2.61 + 2.66 = 2.56
CP add patron category = 1.86 + 2.03 + 2.05 + 2.08 + 2.11 = 2.03
CP add patron = 2.85 + 2.39 + 2.43 + 2.50 + 2.49 = 2.53
CP add biblio = 0.96 + 1.05 + 0.78 + 0.92 + 0.83 = 0.91
CP add items = 10.58 + 9.93 + 10.56 + 9.70 + 9.87 = 10.13
CP checkout = 7.08 + 7.22 + 6.66 + 6.57 + 6.81 = 6.87
CP checkin = 6.16 + 6.25 + 5.81 + 6.02 + 6.18 = 6.08
v3.20.00 under Plack
CP main = 3.33 + 2.66 + 2.52 + 2.61 + 2.43 = 2.71
CP add patron category = 2.87 + 2.74 + 2.49 + 2.76 + 2.54 = 2.68
CP add patron = 2.44 + 2.30 + 2.24 + 2.46 + 2.32 = 2.35
CP add biblio = 0.93 + 0.90 + 0.93 + 0.90 + 0.77 = 0.89
CP add items = 9.81 + 9.29 + 9.96 + 9.62 + 10.71 = 9.88
CP checkout = 7.21 + 6.84 + 6.93 + 6.67 + 6.77 = 6.88
CP checkin = 5.57 + 6.14 + 6.48 + 5.61 + 5.93 = 5.95
master (3.22) under Plack
CP main = 3.99 + 2.84 + 3.21 + 3.14 + 3.41 = 3.32
CP add patron category = 3.06 + 3.02 + 3.74 + 3.55 + 3.32 = 3.34
CP add patron = 3.13 + 2.91 + 3.32 + 3.26 + 2.92 = 3.11
CP add biblio = 1.01 + 1.14 + 1.10 + 0.94 + 0.78 = 0.99
CP add items = 14.14 + 14.61 + 12.08 + 12.16 + 12.39 = 13.08
CP checkout = 10.00 + 8.99 + 8.98 + 8.72 + 9.71 = 9.28
CP checkin = 7.70 + 7.36 + 7.88 + 7.63 + 7.19 = 7.55
In a table
| x | 3.16 | 3.18 | 3.20 | Master (3.22) | 3.16 Plack | 3.18 Plack | 3.20 Plack | Master (3.22) Plack |
|---|---|---|---|---|---|---|---|---|
| main | 3.36 | 4.68 | 4.86 | 5.17 | 2.45 | 2.56 | 2.71 | 3.32 |
| add patron category | 4.04 | 6.2 | 4.49 | 7.32 | 1.95 | 2.03 | 2.68 | 3.34 |
| add patron | 4.24 | 6.59 | 5.89 | 7.27 | 2.27 | 2.53 | 2.35 | 3.11 |
| add items | 12.91 | 17.63 | 13.54 | 20.13 | 9.36 | 10.13 | 9.88 | 13.08 |
| checkout | 11.44 | 15.88 | 15.99 | 17.44 | 6.2 | 6.87 | 6.88 | 9.28 |
| checkin | 9.48 | 14.4 | 14.4 | 15.6 | 5.57 | 6.08 | 5.95 | 7.55 |
And the handsome graph
Quick conclusion/interpretation
- We are almost reaching the same performance with Plack than the 3.16, without Plack, which is a bit scary.
- The gain introduced in 3.20 because of Bug 13690 is considerably lost in 3.22
- There is worrying gap between 3.20 and master for the "add items" step (to investigate...)
- The difference in the "add patron category" step is certainly caused by bug 14836 - Move the patron categories related code to Koha::PatronCategories (to confirm)
- ... ?
Koha 3.8 on Raspberry Pi 1 vs. Koha 3.22 on Raspberry Pi 2
This was done using benchmark_staff.pl. The 3.8 figures are taken from the tests I did mid-2012. The 3.22 figures were done with a self-built package of 3.22 the day after release.
Hardware:
- RPi1 had 1 CPU and 256MB RAM
- RPi2 has 4 CPU and 1024MB RAM
Results:
- Green means 3.22 with Plack is faster than 3.8 with Plack.
- Yellow means 3.22 with Plack is slower than 3.8 with Plack, but still much faster than without Plack. Compared to 3.8 with Plack we lose between 1/3 and 1/2 in performance.
- Red means 3.22 with Plack is much slower than 3.8 with Plack, and not much faster than 3.8 or 3.22 without Plack. Compared to 3.8 with Plack we lose almost everything.
Considering that the hardware used for the test of 3.22 is much better, this is even weirder. The 4 CPU cores of the RPi2 only get used in the last test, and there we have only about 2.5x performance gain with Plack left.
| NoPlack | Plack | ||||
| RPi1 3.8 | RPi2 3.22 | RPi1 3.8 | RPi2 3.22 | ||
| Step 1: staff client main page | 0.049 | 0.076 | 0.462 | 0.582 | pages/sec |
| Step 2: catalog detail page | 0.045 | 0.059 | 0.523 | 0.235 | biblios/sec |
| Step 3: catalogue search | 0.047 | 0.055 | 0.539 | 0.086 | biblios/sec |
| Step 4: patron detail page | 0.049 | 0.066 | 0.518 | 0.362 | borrowers/sec |
| Step 5: patron search page | 0.047 | 0.068 | 0.531 | 0.386 | borrowers/sec |
| Step 6a circulation (checkouts) | 0.049 | 0.060 | 0.478 | 0.276 | checkouts/sec |
| Step 6b circulation (checkins) | 0.046 | 0.069 | 0.504 | 0.390 | checkins/sec |
| all transactions at once | 0.014 | 0,183 | 0.522 | 0.469 | operations/sec |