Integrating Travis CI with your Plone add-ons hosted on GitHub

Update 15/9/2012: Kudos for Asko Soukka who has developed an alternative method of installing Plone using the old good universal installer that reduces the amount of time needed by half. So go and read his post instead of loosing your time with mine.

I took me a little bit but, with the help of Mikko and Martin, I've got a couple of add-ons running tests with Travis CI.

Before setting up Travis CI, you have to make some changes to the Setup Script of your package.

In my case, my add-on package only works for Plone versions 4.1 and later, so I have added Products.CMFPlone as a dependency:

    …
    install_requires=[
        'setuptools',
        'Products.CMFPlone>=4.1',
        ],
    extras_require={
        'test': ['plone.app.testing'],
        },
    …

Products.CMFPlone contains a cut down feature set: just the things I need to run my tests.

Setting up Travis CI is pretty easy: just sign in and activate your GitHub Service Hook

Now, you need to configure your Travis CI build with a .travis.yml file in the root of your repo:

In my case I'm running tests for Plone's latest stable release (4.2 at I write this post) on top of Python 2.7.

Let's take a look at the travis.cfg buildout configuration file:

The main issue I experimented on my first tests was timeouts, so I have a couple of tricks here for you: first, we are extending the standard Plone testing buildout configuration that includes most declarations for us and takes care of always running the latest stable version; we are only going to use the test part. You need to add the package-extras just if your add-on is using plone.app.testing on the test option in extras_require of your package declaration as mentioned above.

zope.globalrequest is needed to run the tests and it was not included on Products.CMFPlone (this is already fixed on Plone's branches for versions 4.2 and 4.3). You may also need to include Pillow in test-eggs; just uncomment it the line.

We also need to add a socket-timeout of 3 seconds (only available on zc.buildout >= 1.5.0) and a list of allow-hosts to download the dependencies. This is pretty important and will avoid timeouts as Travis CI has hard time limits and timeouts are between 10 and 15 minutes for test suite runs (1).

Last, we have to replace the eggs option on the test part; we need to do this because we don't want to include neither Plone or plone.app.upgrade on the tests.

Finally, you can add a Status Image with a link back to the result of your last build on your README.txt file:

.. image:: https://secure.travis-ci.org/collective/your.package.png
    :target: http://travis-ci.org/collective/your.package

To run the tests you only need to make a push to your GitHub repo. Easy, isn't it?

For a live example of all I mentioned above, take a look at the collective.prettydate package.

Travis CI is really easy to set up and fun to use; I strongly recommend it and, if you like it, please show your love donating.

Ray Bradbury in memoriam

The Illustrated Man turned in the moonlight. He turned again… and again… and again…
Ray Bradbury (The Illustrated Man, 1951)

Ray Bradbury, author of Fahrenheit 451 and one of the greatest science fiction writers of the 20th century, died at 91 two days ago.

I leave with you, as an homage, a fragment of one the most beautiful stories written by him.

Kaleidoscope (fragment)

[…]

The many good-bys. The short farewells. And now the great loose brain was disintegrating. The components of the brain which had worked so beautifully and efficiently in the skull case of the rocket ship firing through space were dying one by one; the meaning of their life together was falling apart. And as a body dies when the brain ceases functioning, so the spirit of the ship and their long time together and what they meant to one another was dying. Applegate was now no more than a finger blown from the parent body, no longer to be despised and worked against. The brain was exploded, and the senseless, useless fragments of it were far scattered. The voices faded and now all of space was silent. Hollis was alone, falling.

They were all alone. Their voices had died like echoes of the words of God spoken and vibrating in the starred deep. There went the captain to the Moon; there Stone with the meteor swarm; there Stimson; there Applegate toward Pluto; there Smith and Turner and Underwood and all the rest, the shards of the kaleidoscope that had formed a thinking pattern for so long, hurled apart.

And I? thought Hollis. What can I do? Is there anything I can do now to make up for a terrible and empty life? If only I could do one good thing to make up for the meanness I collected all these years and didn’t even know was in me! But there’s no one here but myself, and how can you do good all alone? You can’t. Tomorrow night I’ll hit Earth s atmosphere.

I’ll burn, he thought, and be scattered in ashes all over the continental lands. I’ll be put to use. Just a little bit, but ashes are ashes and they’ll add to the land.

He fell swiftly, like a bullet, like a pebble, like an iron weight, objective, objective all of the time now, not sad or happy or anything, but only wishing he could do a good thing now that everything was gone, a good thing for just himself to know about.

When I hit the atmosphere, I’ll burn like a meteor.

“I wonder,” he said, “if anyone’ll see me?”



The small boy on the country road looked up and screamed. “Look, Mom, look! A falling star!”

The blazing white star fell down the sky of dusk in Illinois. “Make a wish,” said his mother. “Make a wish.”

(1949)

Photo: Eneas.

Running pep8 before any commit on Git

Readability counts.

The Zen of Python

One of the things that made me choose Python as a programming language in the first place was its readability.

PEP 8 —the Style Guide for Python Code— gives coding conventions for the Python code and pep8 is a tool to check your code against these conventions.

I use gedit as my editor and I have installed the developer plugins to check my code against PEP 8 every time I save a file but, as not everyone does this, Érico asked me today about a way to enforce this practice.

I started searching the web and I have compiled (from 1, 2 and 3) a nice solution using a pre-commit hook with Git (this is possible also with Subversion, but I'm not pretty interested on it right now):

First you need to be sure you are running Git version 1.7.1 or later, and that you have pep8 installed in your system (check the package documentation).

Create a directory to store the hooks globally:

mkdir -p ~/.git_template/hooks

Tell Git all new repositories you create or clone will use this directory for templates:

git config --global init.templatedir '~/.git_template'

Put the following script in the ~/.git_template/hooks directory:


Make the file executable:

chmod +x ~/.git_template/hooks/pre-commit

If you want to use this hook on an existing repository all you have to do is reinitialize it:

git init

Now the pre-commit hook script lives in the .git/hooks directory of your repository.

Test it trying to commit some changes: if the files you are trying to commit comply with PEP 8 (excepting the list of errors or warnings to ignore), the commit will be done as usual; if there are any issues, the commit will be aborted until you fix them.

Feel free to modify the list of errors and warnings to ignore, globally or from project to project, to fit your personal needs.

Remember PEP 8:

A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is most important.

But most importantly: know when to be inconsistent -- sometimes the style guide just doesn't apply. When in doubt, use your best judgment. Look at other examples and decide what looks best. And don't hesitate to ask!

Two good reasons to break a particular rule:

1. When applying the rule would make the code less readable, even for someone who is used to reading code that follows the rules.
2. To be consistent with surrounding code that also breaks it (maybe for historic reasons) — although this is also an opportunity to clean up someone else's mess (in true XP style).

Robot Framework and SeleniumLibrary for Plone developers

Update 05/06/2012: Added a tip for handling overlays and a reference to datakurre's work.

Let's face it: zope.testbrowser sucks for functional and acceptance tests: tests are boring to write, hard to debug when they fail, and there's no support for JavaScript.

The latest is becoming more important every day, as Plone codebase now includes almost 30% of JavaScript code according to Ed Manlove (and Ohloh).

There are some alternatives that help on solving these issues and Selenium —a portable software testing framework for web applications— stands among the best.

Some years ago I started using Selenium to make some functional tests, but I abandoned the task because it was a little bit slow and boring.

During the sprints held after the Plone Conference 2011 in San Francisco, Godefroid Chapelle introduced us to Robot Framework —a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD)— and SeleniumLibrary —a Robot Framework test library that uses the Selenium web testing tool internally.

Writing tests with Robot Framework is pretty easy: you can create tests by using RIDE —a light-weight and intuitive editor for Robot Framework test case files— or just by using your favorite text editor.

A test case is made with a list of keywords and you can create more keywords using the same syntax used for creating the test cases; you can use natural language and you can reuse your test code. This way you start pretty slow but you become more and more productive with every new keyword you add to your test suites.

After coming back home I started adding some very basic functional tests to some of the packages we have developed lately.

I'm not interested on writing a Robot Framework tutorial; there are many on the web (How to use Robot Framework with the Selenium Library is a pretty good one; part 2 and Extending Robot Framework to check emails, are also available).

The purpose of this post is to help other Plone developers on getting Robot Framework up and running and to share some tricks I've learned over the last few months.

Don't panic

Basically all you need to start playing with Robot Framework and SeleniumLibrary in Plone is a buildout configuration, a small resource file including some basic keywords for Plone, a couple of helper files and some test suites.

You can take a look at the test suites written during the sprint in the 4.1-robot branch of Plone's buildout.coredev on GitHub, and you can grab the basic files from there (pybot.cfg and the whole acceptance-tests and templates directories); we are going to modify them a little bit in a minute.

You will also need to download and install the Selenium IDE Plugins —an integrated development environment for Selenium scripts implemented as a Firefox extension— in your Firefox browser (sorry, but I don't know if there are other browser options available at the moment).

First, here is the simplified buildout configuration I've been working on based on the one created by Godefroid:

[buildout]
extends = buildout.cfg
parts += plonesite robot selenium library-settings

[versions]
selenium-server = 2.22.0

[plonesite]
recipe = collective.recipe.plonesite
profiles = my.package:default

[robot]
recipe = zc.recipe.egg
eggs =
    robotframework
    robotframework-seleniumlibrary
entry-points = pybot=robot:run_cli rebot=robot:rebot_cli
arguments = sys.argv[1:]

[selenium]
recipe = hexagonit.recipe.download
download-only = true
url = http://selenium.googlecode.com/files/selenium-server-standalone-${versions:selenium-server}.jar
filename = selenium-server.jar

[library-settings]
recipe = collective.recipe.template
input = templates/library-settings.txt.in
output = ${buildout:directory}/acceptance-tests/library-settings.txt

We extend our standard development configuration by adding four parts: plonesite, that creates a new Plone site, if there is none, and runs the profiles listed (my.package:default); robot, that downloads the robotframework and robotframework-seleniumlibrary eggs, and creates the commands needed to run the tests (yes, this is 2012 and they are still unaware of egg entry points); selenium, that downloads the Selenium standalone server; and, finally, library-settings, that is used to initialize some variables used by the tests, it takes a template file (templates/library-settings.txt.in) and replaces buildout variables with their values creating a new file (acceptance-tests/library-settings.txt).

Please note the use of a ${versions:selenium-server} variable in the selenium part of the configuration; this helps us on keeping it up to date just like with any other egg.

Let's get started: add the files you downloaded to the buildout you want to test. Run bin/buildout -c pybot.cfg and you will find a couple of new scripts inside your bin directory: pybot and rebot.

Start your instance as usual using bin/instance fg and, on another terminal window, run bin/pybot acceptance-tests. Sit comfortably; let the show begin…

After a few seconds a couple of Firefox windows will appear: the first one is the RemoteRunner which is required to control the Plone site running in the second window.

The tests will run and you will get a complete report of their results as a couple of HTML files: report.html includes summary information, test statistics and details; log.html includes a complete log of all tests executed.

In case of error, log.html includes detailed information about it, the source code of the page where the error ocurred and a screenshot of it.

Test Report

Test Log

Share and enjoy

And now for something completely different… the tips and tricks.

What to include in your .gitignore file

Most of our packages live on GitHub and I like include a list of objects to be ignored from version control using a .gitignore file; the following is what I use to add in packages using Robot Framework and SeleniumLibrary tests:

library-settings.txt
log.html
output.xml
report.html
selenium* 

Running a single test suite

If you want to run only one test suite you can do something like this:

bin/pybot -s test_suite acceptance-tests

Handling overlays

Suppose you have to wait for an overlay window to show up in the screen. By default, a page load is expected to happen whenever a link or image is clicked, or a form submitted. In this case we pass the don't wait argument to the keyword.

Delete Item
    [Arguments]  ${title}

    Click Link  ${title}
    Click Link  Delete  don't wait
    Wait Until Page Contains  Do you really want to delete this item?
    Click Button  Delete

As there is no Wait Until Page Does Not Contains nor Wait Until Page Does Not Contains Element keywords, you will have to use something like this in case you want to know if an overlay was closed:

    Wait Until Keyword Succeeds  1  5  Page Should Not Contain  ${text}

The trick here is the Wait Until Keyword Succeeds keyword. If the specified keyword does not succeed within timeout, this keyword fails and waits lapse of time before trying to run the keyword again.

Uploading files and images

This is a typical test case: let's say you developed a new content type that includes a file or image field and you want to test it. A basic test case could be something similar to this one:

*** Settings ***

Resource  plone.txt
Suite Setup  Setup

*** Variables ***

${link_locator} =  a#file
${input_identifier} =  input#file_file

*** Test cases ***

Test Add Audio File
    Goto Homepage
    Add File  ${PATH_TO_TEST_FILES}/test.mp3

*** Keywords ***

Setup
    Log In  admin  admin

Add File
    [arguments]  ${file}

    Open Add New Menu
    Click Link  css=${link_locator}
    Page Should Contain  Add File
    Choose File  css=${input_identifier}  ${file}
    Click Button  Save
    Page Should Contain  Changes saved

The variable ${PATH_TO_TEST_FILES} could be declared in your library-settings.txt.in file as something like this (please note theres are 2 spaces after the equal sign):

*** Variables ***

${dollar}{PATH_TO_TEST_FILES} =  ${buildout:directory}/src/my/package/tests

The trick here is the use of a ${dollar} variable that is defined in the
library-settings part of our pybot.cfg buildout configuration as:

[library-settings]
…
dollar = $

In case you need to upload an image you will replace the CSS selectors ${link_locator} and ${input_identifier} with a#image and input#image_file respectively.

So long, and thanks for all the fish

Robot Framework and SeleniumLibrary makes your life much easier and fun when writing functional and acceptance tests: you become more productive with the addition of every new keyword and debugging failures is child's play with the help of page source code and screenshots.

The main drawback at the moment is that we lack a good resource file including more keywords covering other Plone features, but that can be solved easily as soon as more people start using Robot Framework and we find a way to collaborate on this.

Another important point that I had forgotten to mention, until Asko Soukka remembered it to me, is that we don't have a concept of layers to set up fixtures and a way to clean up global state of the Plone site. This is particularly useful if you want to return your site to its original state without having to remove all changes made by some test case. Asko has made some advances on that lately.

Adding test users programmatically using collective.recipe.plonesite

I'm writing some functional tests for a Plone project and I need to add a group of test users every time I create a test site.

collective.recipe.plonesite is a cool Buildout recipe that enables you to create and update a Plone site as part of a buildout run.

I added the following lines to my buildout configuration:

parts =
    …
    plonesite

[plonesite]
recipe = collective.recipe.plonesite
profiles = my.project:default
post-extras = ${buildout:directory}/acceptance-tests/add_test_users.py

The code of the add_test_users.py script is pretty straightforward:

"""This script will add a number of test users to a Plone site.

You can used it in post-extras option of collective.recipe.plonesite. It will
be evaluated after running QuickInstaller and GenericSetup profiles.

@param portal: The Plone site as defined by the site-id option
"""

import logging
logger = logging.getLogger('collective.recipe.plonesite')

test_users = [
    # (username, password, group),
    ('username1', 'password1', 'group1'),
    ('username2', 'password2', 'group2'),
    ('username3', 'password3', 'group3'),
    ]

for username, password, group in test_users:
    if username not in portal.acl_users.getUserIds():
        try:
            portal.portal_registration.addMember(username, password)
            portal.portal_groups.addPrincipalToGroup(username, group)
        except ValueError:
            logger.warn('The login name "%s" is not valid.' % username)
        except KeyError:
            logger.warn('The group "%s" is not valid.' % group)

Enjoy!

(Next time I'll give you my first impressions on SeleniumLibrary, a web testing library for Robot Framework, I'm using to write the tests.)

Setting the right permissions on your blobstorage directory

Have you ever been annoyed by a message saying that the blobstorage directory of your instance has an insecure mode setting?

That happens to me all the time, so today I spent a couple of minutes trying to figure out how to fix it.

In ZODB/blob.py we have the following:

class FilesystemHelper:
    # Storages that implement IBlobStorage can choose to use this
    # helper class to generate and parse blob filenames.  This is not
    # a set-in-stone interface for all filesystem operations dealing
    # with blobs and storages needn't indirect through this if they
    # want to perform blob storage differently.

    …

    def create(self):
        if not os.path.exists(self.base_dir):
            os.makedirs(self.base_dir, 0700)
            log("Blob directory '%s' does not exist. "
                "Created new directory." % self.base_dir)
        if not os.path.exists(self.temp_dir):
            os.makedirs(self.temp_dir, 0700)
            log("Blob temporary directory '%s' does not exist. "
                "Created new directory." % self.temp_dir)

        if not os.path.exists(os.path.join(self.base_dir, LAYOUT_MARKER)):
            layout_marker = open(
                os.path.join(self.base_dir, LAYOUT_MARKER), 'wb')
            layout_marker.write(self.layout_name)
        else:
            layout = open(os.path.join(self.base_dir, LAYOUT_MARKER), 'rb'
                          ).read().strip()
            if layout != self.layout_name:
                raise ValueError(
                    "Directory layout `%s` selected for blob directory %s, but "
                    "marker found for layout `%s`" %
                    (self.layout_name, self.base_dir, layout))

    def isSecure(self, path):
        """Ensure that (POSIX) path mode bits are 0700."""
        return (os.stat(path).st_mode & 077) == 0

    def checkSecure(self):
        if not self.isSecure(self.base_dir):
            log('Blob dir %s has insecure mode setting' % self.base_dir,
                level=logging.WARNING)

Then, the only thing you need to do is run chmod 700 var/blobstorage (owner can read, write and execute) in your installation directory.

Why this directory is created with a different setting (755) is a mystery to solve another day.