Benchmark for 3.22

From Koha Wiki

Jump to: navigation, search
Home
Koha > Technical > Development

Between Koha 3.16 and 3.18, users faced some performance issues.

To understand, the code has been benchmarked, see bug 13690, especially the comment 2.

The 3.22 release is getting close and it would be great to know what users can expect in term of performance.

To get these values, the selenium tests on bug 13691 have been used.

Contents

Raw data

v3.16.00

CP main = 3.37 + 3.28 + 3.40 + 3.40 + 3.37 = 3.36

CP add patron category = 4.12 + 4.23 + 4.03 + 3.75 + 4.09 = 4.04

CP add patron = 4.15 + 4.19 + 4.22 + 4.46 + 4.17 = 4.24

CP add biblio = 1.00 + 1.30 + 0.84 + 0.79 + 0.87 = 0.96

CP add items = 13.89 + 12.80 + 12.68 + 12.69 + 12.47 = 12.91

CP checkout = 11.92 + 11.39 + 11.45 + 11.09 + 11.34 = 11.44

CP checkin = 9.74 + 9.65 + 10.01 + 8.97 + 9.04 = 9.48

v3.18.00

CP main = 4.64 + 4.65 + 4.69 + 4.79 + 4.61 = 4.68

CP add patron category = 6.06 + 5.96 + 6.75 + 6.26 + 5.99 = 6.20

CP add patron = 6.76 + 6.45 + 6.59 + 6.65 + 6.48 = 6.59

CP add biblio = 1.06 + 1.06 + 1.13 + 1.09 + 1.11 = 1.09

CP add items = 18.93 + 17.03 + 17.50 + 17.26 + 17.41 = 17.63

CP checkout = 16.48 + 16.80 + 15.52 + 15.20 + 15.42 = 15.88

CP checkin = 16.53 + 13.92 + 13.61 + 13.85 + 14.10 = 14.40

v3.20.00

CP main = 4.63 + 4.91 + 4.83 + 4.89 + 5.05 = 4.86

CP add patron category = 4.49 + 4.56 + 4.58 + 4.45 + 4.39 = 4.49

CP add patron = 6.03 + 5.88 + 5.90 + 5.94 + 5.71 = 5.89

CP add biblio = 2.17 + 1.03 + 0.86 + 1.24 + 0.98 = 1.26

CP add items = 13.86 + 13.55 + 13.37 + 13.56 + 13.38 = 13.54

CP checkout = 16.03 + 15.86 + 16.06 + 15.91 + 16.08 = 15.99

CP checkin = 10.22 + 9.66 + 10.18 + 10.03 + 10.07 = 10.03

master (3.22 f25fe6dd Fri Now 6 11:21:56 2015)

CP main = 4.98 + 5.13 + 5.16 + 5.46 + 5.14 = 5.17

CP add patron category = 7.32 + 7.22 + 7.58 + 7.17 + 7.30 = 7.32

CP add patron = 7.39 + 7.13 + 7.36 + 7.38 + 7.11 = 7.27

CP add biblio = 1.41 + 1.14 + 0.91 + 0.94 + 1.11 = 1.10

CP add items = 19.63 + 20.15 + 20.61 + 19.94 + 20.30 = 20.13

CP checkout = 17.06 + 17.69 + 17.77 + 17.02 + 17.68 = 17.44

CP checkin = 16.13 + 15.90 + 14.95 + 15.29 + 15.74 = 15.60

v3.16.00 under Plack

CP main = 2.63 + 2.21 + 2.67 + 2.55 + 2.18 = 2.45

CP add patron category = 2.01 + 1.91 + 1.98 + 1.82 + 2.05 = 1.95

CP add patron = 2.19 + 2.33 + 2.21 + 2.40 + 2.22 = 2.27

CP add biblio = 1.03 + 0.99 + 0.79 + 0.89 + 1.03 = 0.95

CP add items = 9.52 + 9.91 + 9.02 + 9.26 + 9.08 = 9.36

CP checkout = 6.25 + 6.12 + 6.44 + 6.13 + 6.08 = 6.20

CP checkin = 5.96 + 5.81 + 5.83 + 5.03 + 5.24 = 5.57

v3.18.00 under Plack

CP main = 2.66 + 2.40 + 2.45 + 2.61 + 2.66 = 2.56

CP add patron category = 1.86 + 2.03 + 2.05 + 2.08 + 2.11 = 2.03

CP add patron = 2.85 + 2.39 + 2.43 + 2.50 + 2.49 = 2.53

CP add biblio = 0.96 + 1.05 + 0.78 + 0.92 + 0.83 = 0.91

CP add items = 10.58 + 9.93 + 10.56 + 9.70 + 9.87 = 10.13

CP checkout = 7.08 + 7.22 + 6.66 + 6.57 + 6.81 = 6.87

CP checkin = 6.16 + 6.25 + 5.81 + 6.02 + 6.18 = 6.08

v3.20.00 under Plack

CP main = 3.33 + 2.66 + 2.52 + 2.61 + 2.43 = 2.71

CP add patron category = 2.87 + 2.74 + 2.49 + 2.76 + 2.54 = 2.68

CP add patron = 2.44 + 2.30 + 2.24 + 2.46 + 2.32 = 2.35

CP add biblio = 0.93 + 0.90 + 0.93 + 0.90 + 0.77 = 0.89

CP add items = 9.81 + 9.29 + 9.96 + 9.62 + 10.71 = 9.88

CP checkout = 7.21 + 6.84 + 6.93 + 6.67 + 6.77 = 6.88

CP checkin = 5.57 + 6.14 + 6.48 + 5.61 + 5.93 = 5.95

master (3.22) under Plack

CP main = 3.99 + 2.84 + 3.21 + 3.14 + 3.41 = 3.32

CP add patron category = 3.06 + 3.02 + 3.74 + 3.55 + 3.32 = 3.34

CP add patron = 3.13 + 2.91 + 3.32 + 3.26 + 2.92 = 3.11

CP add biblio = 1.01 + 1.14 + 1.10 + 0.94 + 0.78 = 0.99

CP add items = 14.14 + 14.61 + 12.08 + 12.16 + 12.39 = 13.08

CP checkout = 10.00 + 8.99 + 8.98 + 8.72 + 9.71 = 9.28

CP checkin = 7.70 + 7.36 + 7.88 + 7.63 + 7.19 = 7.55

In a table

benchmark (number of seconds per step, less is better)
x 3.16 3.18 3.20 Master (3.22) 3.16 Plack 3.18 Plack 3.20 Plack Master (3.22) Plack
main 3.36 4.68 4.86 5.17 2.45 2.56 2.71 3.32
add patron category 4.04 6.2 4.49 7.32 1.95 2.03 2.68 3.34
add patron 4.24 6.59 5.89 7.27 2.27 2.53 2.35 3.11
add items 12.91 17.63 13.54 20.13 9.36 10.13 9.88 13.08
checkout 11.44 15.88 15.99 17.44 6.2 6.87 6.88 9.28
checkin 9.48 14.4 14.4 15.6 5.57 6.08 5.95 7.55

And the handsome graph

Benchmark 316322.png

Quick conclusion/interpretation

  • We are almost reaching the same performance with Plack than the 3.16, without Plack, which is a bit scary.
  • The gain introduced in 3.20 because of Bug 13690 is considerably lost in 3.22
  • There is worrying gap between 3.20 and master for the "add items" step (to investigate...)
  • The difference in the "add patron category" step is certainly caused by bug 14836 - Move the patron categories related code to Koha::PatronCategories (to confirm)
  • ... ?


Koha 3.8 on Raspberry Pi 1 vs. Koha 3.22 on Raspberry Pi 2

This was done using benchmark_staff.pl. The 3.8 figures are taken from the tests I did mid-2012. The 3.22 figures were done with a self-built package of 3.22 the day after release.

Hardware:

  • RPi1 had 1 CPU and 256MB RAM
  • RPi2 has 4 CPU and 1024MB RAM

Results:

  • Green means 3.22 with Plack is faster than 3.8 with Plack.
  • Yellow means 3.22 with Plack is slower than 3.8 with Plack, but still much faster than without Plack. Compared to 3.8 with Plack we lose between 1/3 and 1/2 in performance.
  • Red means 3.22 with Plack is much slower than 3.8 with Plack, and not much faster than 3.8 or 3.22 without Plack. Compared to 3.8 with Plack we lose almost everything.

Considering that the hardware used for the test of 3.22 is much better, this is even weirder. The 4 CPU cores of the RPi2 only get used in the last test, and there we have only about 2.5x performance gain with Plack left.


NoPlack Plack
RPi1 3.8 RPi2 3.22 RPi1 3.8 RPi2 3.22
Step 1: staff client main page 0.049 0.076 0.462 0.582 pages/sec
Step 2: catalog detail page 0.045 0.059 0.523 0.235 biblios/sec
Step 3: catalogue search 0.047 0.055 0.539 0.086 biblios/sec
Step 4: patron detail page 0.049 0.066 0.518 0.362 borrowers/sec
Step 5: patron search page 0.047 0.068 0.531 0.386 borrowers/sec
Step 6a circulation (checkouts) 0.049 0.060 0.478 0.276 checkouts/sec
Step 6b circulation (checkins) 0.046 0.069 0.504 0.390 checkins/sec
all transactions at once 0.014 0,183 0.522 0.469 operations/sec
Personal tools