Let's face it: zope.testbrowser sucks for functional and acceptance tests: tests are boring to write, hard to debug when they fail, and there's no support for JavaScript.
The latest is becoming more important every day, as Plone codebase now includes almost 30% of JavaScript code according to Ed Manlove (and Ohloh).
There are some alternatives that help on solving these issues and Selenium —a portable software testing framework for web applications— stands among the best.
Some years ago I started using Selenium to make some functional tests, but I abandoned the task because it was a little bit slow and boring.
During the sprints held after the Plone Conference 2011 in San Francisco, Godefroid Chapelle introduced us to Robot Framework —a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD)— and SeleniumLibrary —a Robot Framework test library that uses the Selenium web testing tool internally.
Writing tests with Robot Framework is pretty easy: you can create tests by using RIDE —a light-weight and intuitive editor for Robot Framework test case files— or just by using your favorite text editor.
A test case is made with a list of keywords and you can create more keywords using the same syntax used for creating the test cases; you can use natural language and you can reuse your test code. This way you start pretty slow but you become more and more productive with every new keyword you add to your test suites.
After coming back home I started adding some very basic functional tests to some of the packages we have developed lately.
I'm not interested on writing a Robot Framework tutorial; there are many on the web (How to use Robot Framework with the Selenium Library is a pretty good one; part 2 and Extending Robot Framework to check emails, are also available).
The purpose of this post is to help other Plone developers on getting Robot Framework up and running and to share some tricks I've learned over the last few months.
Don't panic
Basically all you need to start playing with Robot Framework and SeleniumLibrary in Plone is a buildout configuration, a small resource file including some basic keywords for Plone, a couple of helper files and some test suites.
You can take a look at the test suites written during the sprint in the 4.1-robot branch of Plone's buildout.coredev on GitHub, and you can grab the basic files from there (pybot.cfg and the whole acceptance-tests and templates directories); we are going to modify them a little bit in a minute.
You will also need to download and install the Selenium IDE Plugins —an integrated development environment for Selenium scripts implemented as a Firefox extension— in your Firefox browser (sorry, but I don't know if there are other browser options available at the moment).
First, here is the simplified buildout configuration I've been working on based on the one created by Godefroid:
[buildout]
extends = buildout.cfg
parts += plonesite robot selenium library-settings
[versions]
selenium-server = 2.22.0
[plonesite]
recipe = collective.recipe.plonesite
profiles = my.package:default
[robot]
recipe = zc.recipe.egg
eggs =
robotframework
robotframework-seleniumlibrary
entry-points = pybot=robot:run_cli rebot=robot:rebot_cli
arguments = sys.argv[1:]
[selenium]
recipe = hexagonit.recipe.download
download-only = true
url = http://selenium.googlecode.com/files/selenium-server-standalone-${versions:selenium-server}.jar
filename = selenium-server.jar
[library-settings]
recipe = collective.recipe.template
input = templates/library-settings.txt.in
output = ${buildout:directory}/acceptance-tests/library-settings.txt
We extend our standard development configuration by adding four parts: plonesite, that creates a new Plone site, if there is none, and runs the profiles listed (my.package:default); robot, that downloads the robotframework and robotframework-seleniumlibrary eggs, and creates the commands needed to run the tests (yes, this is 2012 and they are still unaware of egg entry points); selenium, that downloads the Selenium standalone server; and, finally, library-settings, that is used to initialize some variables used by the tests, it takes a template file (templates/library-settings.txt.in) and replaces buildout variables with their values creating a new file (acceptance-tests/library-settings.txt).
Please note the use of a ${versions:selenium-server} variable in the selenium part of the configuration; this helps us on keeping it up to date just like with any other egg.
Let's get started: add the files you downloaded to the buildout you want to test. Run bin/buildout -c pybot.cfg and you will find a couple of new scripts inside your bin directory: pybot and rebot.
Start your instance as usual using bin/instance fg and, on another terminal window, run bin/pybot acceptance-tests. Sit comfortably; let the show begin…
After a few seconds a couple of Firefox windows will appear: the first one is the RemoteRunner which is required to control the Plone site running in the second window.
The tests will run and you will get a complete report of their results as a couple of HTML files: report.html includes summary information, test statistics and details; log.html includes a complete log of all tests executed.
In case of error, log.html includes detailed information about it, the source code of the page where the error ocurred and a screenshot of it.
 |
| Test Report |
 |
| Test Log |
Share and enjoy
And now for something completely different… the tips and tricks.What to include in your .gitignore file
Most of our packages live on GitHub and I like include a list of objects to be ignored from version control using a .gitignore file; the following is what I use to add in packages using Robot Framework and SeleniumLibrary tests:library-settings.txt log.html output.xml report.html selenium*
Running a single test suite
If you want to run only one test suite you can do something like this:bin/pybot -s test_suite acceptance-tests
Handling overlays
Suppose you have to wait for an overlay window to show up in the screen. By default, a page load is expected to happen whenever a link or image is clicked, or a form submitted. In this case we pass the don't wait argument to the keyword.
Delete Item
[Arguments] ${title}
Click Link ${title}
Click Link Delete don't wait
Wait Until Page Contains Do you really want to delete this item?
Click Button Delete
As there is no Wait Until Page Does Not Contains nor Wait Until Page Does Not Contains Element keywords, you will have to use something like this in case you want to know if an overlay was closed:
Wait Until Keyword Succeeds 1 5 Page Should Not Contain ${text}
The trick here is the Wait Until Keyword Succeeds keyword.
If the specified keyword does not succeed within timeout, this keyword fails and waits lapse of time before trying to run the keyword again.
Uploading files and images
This is a typical test case: let's say you developed a new content type that includes a file or image field and you want to test it. A basic test case could be something similar to this one:
*** Settings ***
Resource plone.txt
Suite Setup Setup
*** Variables ***
${link_locator} = a#file
${input_identifier} = input#file_file
*** Test cases ***
Test Add Audio File
Goto Homepage
Add File ${PATH_TO_TEST_FILES}/test.mp3
*** Keywords ***
Setup
Log In admin admin
Add File
[arguments] ${file}
Open Add New Menu
Click Link css=${link_locator}
Page Should Contain Add File
Choose File css=${input_identifier} ${file}
Click Button Save
Page Should Contain Changes saved
The variable ${PATH_TO_TEST_FILES} could be declared in your library-settings.txt.in file as something like this (please note theres are 2 spaces after the equal sign):
*** Variables ***
${dollar}{PATH_TO_TEST_FILES} = ${buildout:directory}/src/my/package/tests
The trick here is the use of a ${dollar} variable that is defined in the library-settings part of our pybot.cfg buildout configuration as:
In case you need to upload an image you will replace the CSS selectors ${link_locator} and ${input_identifier} with a#image and input#image_file respectively.[library-settings] … dollar = $
So long, and thanks for all the fish
Robot Framework and SeleniumLibrary makes your life much easier and fun when writing functional and acceptance tests: you become more productive with the addition of every new keyword and debugging failures is child's play with the help of page source code and screenshots.The main drawback at the moment is that we lack a good resource file including more keywords covering other Plone features, but that can be solved easily as soon as more people start using Robot Framework and we find a way to collaborate on this.
Another important point that I had forgotten to mention, until Asko Soukka remembered it to me, is that we don't have a concept of layers to set up fixtures and a way to clean up global state of the Plone site. This is particularly useful if you want to return your site to its original state without having to remove all changes made by some test case. Asko has made some advances on that lately.
No comments:
Post a Comment