Automated testing tools

As part of a Bachelor of Information Technology capstone project I have investigated installing, writing tests and running test for 5 automated functional/regression testing tools.

The purpose of this was to compare each of them against 5 criteria to find the most suitable tool for using in the Koha test suite.

The 5 criteria are:

• Time to install(in minutes)
  
• Time it took to write the test(in minutes)
  
• Time to run the test(in seconds)
  
• Reliability - does the test consistently pass or are its outputs inconsistent
• Readability of the test scripts (this is a subjective measurement on a scale of 1-10 where 1 is readability sufficient for a non-developer to understand what the test is doing measured by testing the script against what my father (who is not a developer) could understand)
  

The 5 testing tools evaluated against these criteria are:

• WebInject
• Sahi
• Watir
• Cucumber
• Selenium (using the Selenium standalone server and Selenium::Remote::Driver module to write Perl test scripts for example t/db_dependent/selenium/basic_workflow.t)

For consistencies sake I wrote the same simple test in each of these testing tools. The test simply invloved logging into the Koha staff intranet and then navigating into the Adminitsrtaion module and creating a library.

Results of the research/experimentation:

Here are the results in tabular format, and below that is in depth explnation of how to install each of the 5 tools, write the tests and run the tests:

Tool	Time to install (for first time user not familiar with the tool)	Time to write test (for first time user not familiar with the tool)	Time to run test	Readability (on scale of 1 to 10 where 1 is easy enough for a non-developer to understand what the test is doing)	Reliability
WebInject	1hr	20 mins	16 seconds	8	Good
Sahi	7 minutes	5 mins	17 seconds	8	Good
Watir	1 hr	5 mins	30 seconds	8	Good
Cucumber	1 hr	1 hour	32 seconds	2 (easy to understand as the Cucumber feature files written for each test are written in a simple english syntax called Gherkin)	Good
Selenium	2 hrs (including troubleshooting)	1.5 hrs (including troubleshooting)	30 seconds	8	Bad (the test was falky in that occasionally it would usccessfully run and on other occasions it would fail due to not being able to find a element on one of these lines: lines 83, 87 or 94)

Best performing testing tool according to the above results: Sahi

Sahi is the best performing testing tool in the study because it has the fastest installation time, test writing time, and has a decent test run time and is reliable to give the same consistent result on each running of the test.

Disclaimer: Now Sahi was used to see how it performed in two further tests which were: creating a patron and cataloguing an item. Sahi performed well in the former however it could not select the dropdown div for 'Koha item type' on the cataloguing page due to the fact it was not a valid input field.

However in all other cases such as where a popup window is loaded Sahi successfully interacts with the window.

Worst performing testing tool according to the above results: Selenium

Selenium is the worst performing testing tool, with the major issue being its flakiness, and the time it takes to write each test due to that assoicated flakiness, because you end up having to return to the test to see if each error where an element cannot be found is valid or not.

Testing tools

WebInject

Writing a WebInject test for Koha

This tool has no record and capture ability to create tests, so it takes longer to create the tests as you have to write them instead.

The tests for WebInject are a series of HTTP get and post requests, this guide is useful for understanding how the tests are written:  http://webinject.blogspot.co.nz/2017/04/posting-web-form-with-webinject.html

As the below sample WebInject test shows the first test case (first executed test step) in the test script is to load the search form of this sample website and to verify that the text 'Seach Recipes' is writen on this page.

The second test case is to submit the search form with the inputted values in this HTTP post request being RecipeName=Shephards Pie, Cuisine= British, MaxPrepTime=2

<testcases repeat="1">

<case

   id="10"
 description1="Get Search Form"
 method="get"
 url="http://webinject-check.azurewebsites.net/Search/SearchForm"
 verifypositive="Search Recipes"

/>

<case

   id="20"
 description1="Submit Search"
 method="post"
 url="http://webinject-check.azurewebsites.net/Search/SearchRecipes"
 postbody="RecipeName=Sheperds%20Pie&Cuisine=British&MaxPrepTime=2&BtnSubmit=Search%20Recipes&__RequestVerificationToken={DATA}"
 verifypositive="Search Confirmation"

/>

</testcases> 

This sample test is from: http://webinject.blogspot.co.nz/2017/04/posting-web-form-with-webinject.html

In writing a WebInject test for creating a library in Koha the first step is to authenticate in the Koha login page, the URL to load for this page on my kohadevbox instance is: http://localhost:8081/cgi-bin/koha/mainpage.pl

I then had to write in descriptions (the text I want to find a match for in the HTTP response). The verifypositive function checks if the description matches the text in the HTTP response.

I based my test on the above sample test and I learnt that I had to write in the values I want inputted (in this case the userid and password) into a field called the postbody. The above sample test has a postbody  field and to make sure the syntax is correct for my postbody value I installed Google Chrome on my Linux Ubuntu 16.04 machine and I used the Chrome Developer Console.

Right clicking on the login form and selecting 'Inspect Element' and clicking on the 'Network' tab I recorded submitting the form and under the Form Data title the postbody of the HTTP request I submitted was shown for me to be able to copy and paste into my test script, see below screenshot.

I followed the same process using the Google Chrome Developer console to create the rest of the library creation test which consisted of loading the Administrative module URL (not navigating there for the limitation discussed below, loading the Library creation form URL, and submitting a HTTP post request containing inputted values for the library form. Below is the final code for the WebInject library creation test:

<testcases repeat="1">

<case

id="10"
description1="Load login page"
method="get"
url="http://localhost:8081/cgi-bin/koha/mainpage.pl"

/> <case

id="20"
description1="Submit login credentials"
method="post"
url="http://localhost:8081/cgi-bin/koha/mainpage.pl"
postbody="koha_login_context=intranet&userid=mika&password=secret&branch"
verifypositive="Intranet"

/> <case

id="30"
description1="Load admin module"
method="get"
url="http://localhost:8081/cgi-bin/koha/admin/admin-home.pl"

/> <case

id="40"
description1="Load library creation form"
method="get"
url="http://localhost:8081/cgi-bin/koha/admin/branches.pl"
postbody="op=add_form"

/> <case

id="50"
description1="Submit library creation form"
method="post"
url="http://localhost:8081/cgi-bin/koha/admin/branches.pl" postbody="op=add_validate&branchcode=test&branchname=test&branchaddress1=&branchaddress2=&branchaddress3=&branchcity=&branchstate=&branchzip=&branchcountry=&branchphone=&branchfax=&branchemail=&branchreplyto=&branchreturnpath=&branchurl=&opac_info=&branchip=&branchnotes="

/>

</testcases> 

Results of running the test

When I ran this I found that all of the tests in this test case passed and clicking on the link to see what was displayed to WebInject on the final step I saw that the library creation form had been displayed and the library named ‘test’ was in the list of libraries created.

Here is how WebInject did against the 5 test criteria:

Time to install: 1 hour (plus another hour to understand how to write the test)

• Test to write the test: 20 minutes (including troubleshooting)
• Time to run the test: 16 seconds
• Readability: 8 (hard to understand what the test is doing, however the description values help make the test code and test output more readible) (scale of 1 to 10 where 1 is easy and 10 is hard))
• Reliability: Yes (once you have got the hang of writing the test correctly)
  

Reflection on the usefulness of WebInject

I found a limitation with WebInject is that you often have to hardcode in a URL to load rather than being able to write in a postbody op value for loading another page from the existing page. The result of this is you are not able to find navigation bugs in navigating around Koha as you would with Selenium in which you write in the button id or xpath to click it in order to load another page, so in that sense Selenium is better at finding potential navigation bugs than WebInject.

Limitation of the test I created

The library creation test I wrote up to authenticate into the staff Intranet and create a library was simple enough for me to write and run it in the tested tools within a relatively short space of time. Due to the simple nature of the test a limitation of my primary research is that I have not utilised the full capabilities of testing tools.

Sahi

Installation 

I downloaded the Sahi .jar file from here: https://sourceforge.net/projects/sahi/?source=typ_redirect

I then installed Sahi by writing in the terminal command (into my Linux machine): java -jar install_sahi_v51_20161005.jar

The rest of the installation process was easy keeping the default selected options.

How to start Sahi

To run Sahi after it was installed in my terminal I went to: <sahi home directory>/userdata/bin and wrote in the terminal command: ./start_dashboard.sh

Creating a Sahi test for Koha

To create the test after that you need to click on a browser (in my case I clicked on Mozilla Firefox) and then a new browser window appears. To utilise the record and capture functionality that Sahi provides to create the test cases you need to type: ctrl+alt + double click.

This brings up the Sahi controller:

Giving the test a name and selecting the record button the test creation was started. Into the web browser I wrote in the url: localhost:8081

A prompt appears informing you that the Sahi bypasses localhost and so it may not work with the URL localhost:8081

Simply click ‘OK’ as I found that Sahi does indeed work with localhost:8081

After clicking the ‘OK’ button the Koha login page appeared, after writing in the test credentials of username ‘mika’ and password’ secret’ I was logged in.

After logging in successfully I navigated to the Administration module, clicked on the ‘Libraries and groups’ button, selected the ‘New Library’ button and filled out and submitted a form then stopped the test.

Results of running the test

Stopping the test and then selecting the ‘Record’ tab of the Sahi controller I wrote in the name of the test, firsttscript.sah I wrote in the starting URL to be localhost:8081 and started the test and it passed, see below screenshot:

• Time to install: 7 minutes
• Time to create test: 5 minutes using the record and capture feature
• Time to run test: 17 seconds
• Readability: 8 (it is not clear what the test is doing except it is clear that it is clicking and setting values
• Reliability: Good
  

Some things to keep in mind with Sahi:

1) Because my test involved authenticating into the Koha staff intranet I had to make sure I was logged out of the Koha staff intranet before running the test otherwise the intranet home page rather than the login page would ppear when the test loaded the url: localhost:8081

2) Before playing the test you need to click the ‘Set’ button (the set button is shown in the above screenshot)

The automatically generated test code for Sahi for creating a library in Koha intranet was:

_setValue(_textbox("userid"), "mika");

_setValue(_password("password"), "secret");

_click(_submit("Login"));

_click(_link("Koha administration"));

_click(_link("Libraries and groups"));

_click(_link("New library"));

_setValue(_textbox("branchcode"), "test");

_setValue(_textbox("branchname"), "test");

_click(_submit("Submit[2]")); 

Reflection on the usefulness of Sahi

• Sahi, like Selenium, and unlike WebInject allows you to create tests which test navigability of buttons in the web application, in other words once the Koha home page was loaded I clicked on the ‘Koha Administration’ button to load the Administrative module. This gives Sahi an edge over WebInject as it covers navigation bugs, which WebInject does not do.
  

• Creating tests with Sahi is very fast, faster than Selenium and WebInject which have to be manually written. Another benefit of Sahi is that it is easy to use, and take very little time to learn how to use it.

Watir

Installation

To install and use Watir you need to install Watir-Webdriver which can be installed on Linux, Mac, and Windows and it can work with Chrome, Firefox and Internet Explorer. Whilst Watir alone only works on Windows testing Internet Explorer.

The installation of Watir was not straightforward , I followed the following steps in a process of troubleshooting to get Watir installed:

1. The first step in installing Watir is to install Ruby.

Installing Ruby in Linux Ubuntu is easy I did the following commands in the root directory of my terminal: sudo apt-get install ruby-full

To check that Ruby was installed correctly I wrote in the command: which ruby-full

This showed that Ruby was installed in: /usr/bin/ruby

To test that ruby was working correctly I created a ruby file (test.rb) to echo out ‘Hello world!’.

I ran the test.rb file and it displayed the ‘Hello world!’ message meaning Ruby is successfully installed.

2. To install Watir-webdriver I wrote in the terminal command: sudo gem install watir-webdriver

3. In my reading online it was not clear if you had to install watir itself as well as the watir-webdriver so I ran the command:sudo apt-get install watir

4. To load the watir-webdriver I brought up the irb shell in the terminal and I wrote in: require watir-webdriver

This threw an undefined variable error, to fix this I experimented with writing watir-webdriver in doublequotes( “), single quotes (‘) and tilda’s (`). However this either returned a error message or ‘false’ meaning the the webdriver was not loaded.

5. After some research I found that I needed to install rubygem first so I exited out of irb and wrote in the command: sudo apt-get install rubygem

Then trying again to load the watir-webdriver in the irb I noticed that when I wrote in:require ‘watir-webdriver’ even though a message appeared stating the command was deprecated it still had the returned value of ‘=> true’ below the deprecation message.

6. Bringing up the Firefox browser using Watir-webdriver to see if the webdriver was indeed working I wrote in the command: browser = Watir::Browser.new :firefox

This threw an error asking for the Mozilla geckodriver to be installed, which I did by writing in the following commands (outside of irb):

wget https://github.com/mozilla/geckodriver/releases/download/v0.16.1/geckodriver-v0.16.1-linux64.tar.gz

sudo sh -c 'tar -x geckodriver -zf geckodriver-v0.16.1-linux64.tar.gz -O > /usr/bin/geckodriver'

sudo chmod +x /usr/bin/geckodriver

rm geckodriver-v0.16.1-linux64.tar.gz

Then starting irb again and wrote in:

require “watir-webdriver”
browser = Watir::Browser.new :firefox

This brought up a new firefox window.

Writing a Watir test for Koha

To learn how Watir works I decided to use a sample test from online and try to run it. I tried running 2 sample tests from these two resources: http://opensourceforu.com/2015/03/automating-browsers-with-ruby-and-watir-webdriver/ and https://stackoverflow.com/questions/45314496/ruby-script-that-runs-list-of-irb-watir-commands

The sample test for the latter resource was named the file test_1.rb and stored in the following directory c:/Documents/Testing tools/.

I changed the test as it was outlined on the site by making the wikipedia page be loaded by the browser, in addition to calling the build method at the end of the test (.build) and storing the browser (browser = TestBrowser.build).

require 'watir-webdriver' require 'irb'

class TestBrowser < Watir::Browser

 def self.build
  browser = new(:firefox)
  browser.goto 'www.wikipedia.org'
  # other setup steps
  browser # return the browser at the end of the method
end

end

IRB.start .build browser = TestBrowser.build 

Then after much experimentation I found that the following commands to run this test (test_1.rb) were:

In the Linux terminal write in: ruby ./test_1.rb

This starts irb, then write in:browser = TestBrowser.build

This a Firefox browser appeared and Wikipedia was loaded.

To write the library creation script for Koha I took the sample file which I had altered above and I wrote in the url to be loaded to be: http://www.localhost:8081

Running the test with the commands: ruby ./test_1.rb  the FireFox browser loaded the Koha intranet login page.

2. The login credentials I wanted to submit were username="mika" and the password "secret1", the code to do this was:

browser.text_field(:id, 'userid').when_present(5).set('mika')

browser.text_field(:id, 'password').when_present(5).set('secret')

browser.button(:id, 'submit').click 

Below is the final Watir test for creating a library.

Results of running this test

To run this test I wrote in the terminal commands: ruby ./create_library.rb browser = TestBrowser.build

The browser automatically logged into the staff intranet, navigated to the administration module’s library creation form. Entered in a library with the name ‘test’ and description ‘test’ and submitted the form. The below screenshot shows that the test was successful.

• Time to install: 1 hour (with troubleshooting)
• Time to create test: 5 minutes
• Time to run test: 30 seconds
• Readability: 8 (it is not clear what the test is doing except it is clear that it is clicking and setting values)
• Reliability: Good

Reflection on the usefulness of Watir

• Good stability - Watir does not have a record and capture functionality and the tests are written in much the same way as Perl selenium tests are written, however unlike Selenium Watir tests are not flaky failing because the page is not fully loaded before trying to locate an element. Watir only looks for an element to interact with after the page is fully loaded.

• Takes longer to write the tests due to not having a record and capture functionality

Cucumber

Installation

1. As with Watir Cucumber uses Ruby and so you need to make sure it is installed by running the terminal command: ruby -v

2. After ensuring that Ruby was installed I installed Cucumber with the command: sudo gem install cucumber

3. Then Gherkin which is a simple english language which is used to write one of the two test files for each test in Cucumber (the Gherkin tests are known as feature files) was installed with the command: sudo gem install gherkin
4. Then Capybara had to be installed with this command: sudo gem install capybara

5. Then Rspec had to be installed with this command:sudo gem install rspec

RSpec is a “Behaviour-Driven Development tool for Ruby programmers. BDD is an approach to software development that combines Test-Driven Development, Domain Driven Design and Acceptance Test-Driven Planning” (http://www.c-sharpcorner.com/UploadFile/b6cb4b/installation-of-cucumber-in-ubuntu)

6. Then I had to install Selenium Webdriver with the command: sudo gem install selenium-webdriver

The Selenium WebDriver is what drives the browser interacting with it as a user would.

Writing a Cucumber test for Koha

The guide I used to learn how to write Cucumber tests was https://semaphoreci.com/community/tutorials/introduction-to-writing-acceptance-tests-with-cucumber. This guide outlined that each test written for Cucumber should have two files, the feature file written in Gherkin which non-developers can easily understand and a step definition file (ruby script implementing the feature file).

The format that the Gherkin feature file is written in is:

Feature: <featurename> As a <usertype> I want <functionality> So that <reason>

Scenario: <scenario name>

When

Then I <action>

Following the same guide I wrote the feature file for the library creation test in Koha:

I wrote a step definition (in ruby) to run to drive the browser for every Given, When, Then command in the feature file. The test itself is built to use the Selenium WebDriver and so it has many similarities to the Perl tests I had written for Selenium. For example at the top of the file a variable named driver is assigned the Selenium webdriver for Firefox browser and the navigate.to function is called to load the localhost:8081 URL.

From there the find_element function is used to find input fields and buttons to either input values into using the send_keys() function or click function to interact with the page.

Unlike the Perl Selenium tests which are headless tests (meaning the browser GUI is not displayed as the test is being run) the Cucumber test does show the browser interaction.

Results of running this test

To run a Cucumber test you run the feature file and as long as the step definition file is in the step definition subdirectory of the directory holding the feature file then the step definition script should be run. To run the test you need to write in the command: cucumber <featurefilename>.feature

I did this for the library creation test and it succeeded:

• Time to install: 1 hour (with troubleshooting)
• Time to create test: 1 hour (this was because with Cucumber you have to write two files for each test a feature file and a step definition. However the main reason that the test took so long to write was because troubleshooting was performed).
• Time to run test: 32 seconds
• Readability: 2 (by having the feature file written in Gherkin (simple english formatted in the format of Given, When, Then) which non-developers can understand this makes Cucumber tests must easier to understand than all of the other tests)
• Reliability: Good
  

Reflection on the usefulness of Cucumber

Cucumber utilises the Selenium WebDriver (note: the Perl Selenium test I wrote was using the Perl Selenium package Selenium::Remote::Driver), but the difference it has from the Perl Selenium::Remote::Driver tests were:

• Cucumber enables part of the test (feature file) to be readable to non-developers this could prove allow librarians in the Koha community to write the feature files in Gherkin which developers could then write the associated Ruby scripts for. This is an advantage over Selenium alone which with Selenium::Remote::Driver has tests which are not as readable to non-developers as they are written in Perl.

• Non headless test so the Firefox browser shows the test being executed, as opposed to the headless Selenium::Remote::Driver tests

• Cucumber has stable and reliable tests were the web driver does not try to find an element before the page is finished loading. This is an issue with Selenium::Remote::Driver Perl tests which is solved by implementing pause() functions.

• By writing two test files for each test it takes slightly longer to create tests in Cucumber than other testing tools, particularly the tools with the record and capture functionality such as Sahi.

Selenium

Installation

I had been working with Selenium::Remote::Driver (which I will refer to as Selenium) for Catalyst in my contract work and so I am well aware of its limitations.

The steps to install Selenium and Selenium::Remote::Driver are:

1. Download the standalone Selenium server
wget https://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.1.jar

2. vim /etc/apt/sources.list.d/firefox.list

3. deb http://packages.linuxmint.com debian import

4. sudo apt-get update

5. sudo apt-get install firefox

6. sudo apt-get install xvfb

7. SELENIUM_PATH=/home/vagrant/selenium-server-standalone-2.53.1.jar

8. Xvfb:1 -screen 0 1024x768x24 2>&1 >/dev/null &

9. DISPLAY=:1 java -jar $SELENIUM_PATH

Or if the command on step 9 does not work then write in the command:

DISPLAY=:1 java -jar selenium-server-standalone-2.53.1.jar

Note:: This starts the Selenium standalone server

10. git clone https://github.com/gempesaw/Selenium-Remote-Driver --branch build/master --single-branch --depth 1

11. cd Selenium-Remote-Driver

12. perl Makefile.PL

13. make

14. make test

15. sudo make install

The Selenium tests I have been writing in Koha exist in the t/db_dependent/selenium/* directory, and are run from the Koha-shell using the following commands:

sudo koha-shell <instancename>
perl <filename>.t

Writing a Selenium test for Koha

The syntax of the library creation test was similar to that of the step definition of the Cucumber tests, not in the sense of matching Given, When, Then statements because Selenium using Selenium::Remote::Driver does not have feature files containing these statements but in the sense of similar functions to interact with the webpage such as the click and send_keys functions.

The reason for this is that is because Cucumber is using Ruby to drive the Selenium WebDriver whereas Selenium::Remote::Driver is simply the Perl package version of WebDriver, in other words the webdriver is essentially the same with the difference being the different programming language.

The below screenshot shows the finished short library creation test Perl script from line 82 to 87 with the rest of the test script being existing code from basic_workflow.t

Note: Most of the Selenium tests I have written are considerably longer than this test.

To run the test I wrote into my Linux terminal:
sudo koha-shell kohadev
perl create_library.t

Results of running this test

Now running the test was problematic because Selenium will sometimes pass and sometimes fail with no code changes being made in between the successful and failing runs of a test.

A common test failure error with Selenium is “An element could not be located on the page using the given search parameters”. Now in the case of the library creation test there were three lines where the test consistently failed and they are lines 83, 87, and 94.  This error is often caused by the fact that the webpage is often not fully loaded before the web driver tries to find the element, see figure.

Therefore a common solution for these errors that I have previously used is implementing the pause() function to force the webdriver to pause for a set about of time (3000 milliseconds or 3 seconds) before looking for the element.

However even after implementing the $driver->pause(3000); function immediately before lines 83, 87 and 94 the test continued to fail on occasion as the below screenshot shows.

However on other occasions after no changes being made to the test or the Koha codebase the test would be successfully executed as the below screenshot shows.

• Time to install: 2 hours (with troubleshooting)
• Time to create test: 1.5 minutes
• Time to run test: 30 seconds
• Readability: 8 (it is not clear what the test is doing except it is clear that it is clicking and setting values)
• Reliability: Bad
  

Reflection on the usefulness of Selenium

The flakiness of Selenium is unique out of these 5 testing tools, the other 4 testing tools investigated in this primary research would reliably pass every time, when the test has been correctly written.