Monday, December 11, 2017

We have been doing health checks wrong in Plone

In the previous post of this blog I was arguing on how to increase the performance of high-traffic Plone sites. As mentioned there, we have different ways to do so: increasing the number of server threads, increasing the number of instances, and increasing both.

Increasing the number of threads is easier and consumes less memory but, as mentioned, is less resilient and can be affected by the Python GIL on multicore servers. Increasing the number of instances, on the other side, will increase the complexity of our stack as we will need to install a load balancer, a piece of hardware or software that distributes the incoming traffic across all the backend instances.

Over the years we have tried different software load balancers depending on the traffic of a site: we use nginx alone as web server and load balancer on smaller sites and we add Varnish as web accelerator and load balancer when the load increases; lately we started using HAProxy again to solve extreme problems on sites being continuously attacked (I'll write about on a different post).

When you use a load balancer you have to do health checking, as the load balancer needs to know when one of the backend instances has become unavailable because is quite busy to answer further requests, has been restarted, is out for maintenance, or is simply dead. And, in my opinion, we have been doing it wrong.

The typical configuration for health checking is to send requests to the same port used by the Zope HTTP server and this has some fundamental problems: First, the Zope HTTP server is slow to answer requests: it can take hundreds of milliseconds to answer even the most simple HEAD request.

To make things worst, the requests that are answered by the Zope HTTP server are the slower ones (content not in ZODB cache, search results, you name it…), as the most common requests are already being served by the intermediate caches. In the tests I made, I found that even a well configured server running a mature code base can take as many as 10 seconds to answer this kind of requests. This is a huge problem as health check requests start queuing and timing out taking perfectly functional instances out of the pool, and making things just worst.

To avoid this problem we normally configure health checks with long intervals (typically 10 seconds) and windows of 3 failures for every 5 checks. And this, of course, creates another problem as the load balancer takes, in our case, up to 30 seconds to discover that an instance has been restarted when using things like Supervisor and its memmon plugin, leading to a lot of 503 Service Unavailable errors in the mean time.

So, it's a complete mess no mater how you analyze it and I needed to find a way to solve it: enters five.z2monitor.

five.z2monitor plugs zc.monitor and zc.z3monitor into Zope 2, enabling another thread and port to handle monitoring. To install it you just need to add something like this into your buildout configuration:

[buildout]
eggs =
    …
    five.z2monitor
zcml =
    …
    five.z2monitor

[instance]

zope-conf-additional =
    <product-config five.z2monitor>
        bind 127.0.0.1:8881
    </product-config>


After running buildout and restarting your instance you can communicate with your Zope server over the new port using different commands, called probes:

$ bin/instance monitor help
Supported commands:
  dbinfo -- Get database statistics
  help -- Get help about server commands
  interactive -- Turn on monitor's interactive mode
  monitor -- Get general process info
  ok -- Return the string 'OK'.
  quit -- Quit the monitor
  zeocache -- Get ZEO client cache statistics
  zeostatus -- Get ZEO client status information


Suppose you want to get the ZEO client cache statistics for this instance; all you have to do is use the following command:

$ bin/instance monitor zeocache main
417554 895451465 435095 900622900 35429160


You can also use the Netcat utility to get the same information:

$ echo 'zeocache main' | nc -i 1 127.0.0.1 8881
417753 896710955 435422 901905068 35467686


It's easy to extend the list of supported commands by writing you own probes; in the list above I have added to the default command set one that I create to know if the server is running or not; it's called "ok", and here is its source code:

import pkg_resources
import logging


logger = logging.getLogger('zc.monitor')


try:
    pkg_resources.get_distribution('zc.monitor')
except pkg_resources.DistributionNotFound:
    logger.info('Monitor server is not available')
else:
    def ok(connection):
        """Return the string 'OK'."""
        connection.write('OK\n')

    import zc.monitor
    zc.monitor.register(ok)
    logger.info('"OK" command for monitor server registered')


We have now a dedicate port and thread that can be used for health checking:

$ echo 'ok' | nc -i 1 127.0.0.1 8881
OK


With this we solved most of the problems I mentioned above: we have faster response time and no queuing; we can decrease the health check interval to a couple of seconds and we are almost sure that a failure is a failure and not just a timeout.

Note we can't use this with nginx, nor Varnish, as their health checks are limited and expect the same port used for HTTP requests; only HAProxy supports this configuration.

So, in the next post I'll show you how to configure HAProxy health checks to use this probe and how to reduce the latency to a couple of milliseconds.

Friday, December 23, 2016

Plone performance: threads or instances?

Recently we had a discussion on the Plone community forum on how to increase performance of Plone-based sites.

I was arguing in favor of instances, because some time ago I read a blog post by davisagli taking about the impact of the Python GIL on performance of multicore servers. Others, like jensens and djay, were skeptical on this argument and told me not to overestimate that.

So, I decide to test this using the production servers of one of our customers.

The site is currently running on 2 different DigitalOcean servers with 4 processors and 8GB RAM; we are using Cloudflare in front of it, and round-robin, DNS-based load balancing.

Prior to my changes, both servers were running with the same configuration:
  • nginx, doing caching of static files and proxy rewrites
  • Varnish, doing caching and URL-based load balancing
  • 4 Plone instances running on ZEO client mode, with 1 thread and 100.000 objects in cache

Both servers where running also a ZEO server on ZRS configuration, one as a master and the other as a slave, doing blob storage replication.

First, here we have some information from this morning, before I made the changes. Here are some graphics I obtained using New Relic on the master:




Here is the same information from the slave:



As you can see, everything is running smoothly: CPU consumption is low and memory consumption is high and… yes, we have an issue with some PhamtomJS processes left behind.

This is what I did later:

  • on the master server, I restarted the 4 instances
  • on the slave server, I changed the configuration of instance1 to use 4 threads and restarted it; I stopped the other 3 instances
I also stopped the memmon Supervisor plugin (just because I had no idea on how much memory the slave server instance will be consuming after the changes), and killed all PhamtomJS processes.

The servers have been running for a couple of hours now and I can share the results. This is the master server:



And this is now the slave:



The only obvious change here is in memory consumption: wow! the sole instance on the slave server is consuming 1GB less than the 4 instances in the master server!

Let's do a little bit more research now. Here we have some information on database activity on the master server (just one instance for the sake of simplicity):



Now here is some similar information for the slave server:



I can say that I was expecting this: there's a lot more activity and the caching is not very well utilized on the slave server (see, smcmahon, that's the beauty of the URL-based load balancing on Varnish demonstrated).

Let's try a different look, now using the vmstat command:



Not many differences here: the CPU is idle most of the time and the interrupts and context switching are almost the same.

Now let's see how much our instances are being used, with the varnishstat command:




Here you can see why there's not too much difference: in fact Varnish is taking care of nearly 90% of the requests and we have only around 3 requests/second hitting the servers.

Let's make another test to see how quickly we are responding the requests using the varnishhist command:



Again, there's almost no difference here.

Conclusion: for our particular case, using threads seems not to affect too much the performance and has a positive impact on memory consumption.

What I'm going to do now is to change the configuration used in production to have 2 instances and 2 threads… why? because restarting a single instance on a server for maintenance purposes would let us without backends during the process if we were using only one instance.

Share and enjoy!

Wednesday, April 30, 2014

Using pep257, a Python docstring style checker, with Buildout

PEP 257 documents the semantics and conventions associated with Python docstrings. According to it:

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.

This is specially useful, for instance, when you are introspecting code and you want to know what a specific method or function does.

pep257 is a Python docstring style checker.

I already filled a feature request on plone.recipe.codeanalysis to add support for it; meanwhile, if you want to use it in your Buildout-based projects, you can add the following to your buildout configuration:

After running bin/buildout you will find a bin/pep257 script.

A typical output will look like this:

(python-2.7)# hvelarde@nanovac (master * u+1) ~/collective/polls 
# bin/pep257 src/
src/collective/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/js_i18n_helper.py:1 at module level:
        D100: Docstring missing
src/collective/polls/js_i18n_helper.py:8 in public class `LegendOthersTranslation`:
        D101: Docstring missing
src/collective/polls/js_i18n_helper.py:14 in public method `render`:
        D102: Docstring missing
src/collective/polls/testing.py:1 at module level:
        D100: Docstring missing
src/collective/polls/testing.py:10 in public class `Fixture`:
        D101: Docstring missing
src/collective/polls/testing.py:14 in public method `setUpZope`:
        D102: Docstring missing
src/collective/polls/testing.py:19 in public method `setUpPloneSite`:
        D102: Docstring missing
src/collective/polls/config.py:1 at module level:
        D100: Docstring missing
src/collective/polls/polls.py:1 at module level:
        D100: Docstring missing
src/collective/polls/polls.py:21 in public class `IPolls`:
        D101: Docstring missing
src/collective/polls/polls.py:24 in public method `recent_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:24 in public method `recent_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:27 in public method `uid_for_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:27 in public method `uid_for_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:30 in public method `poll_by_uid`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:30 in public method `poll_by_uid`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:33 in public method `voters_in_a_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:33 in public method `voters_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:36 in public method `voted_in_a_poll`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:36 in public method `voted_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D401: First line should be imperative: 'i', not 'is'
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:39 in public method `allowed_to_edit`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:42 in public method `allowed_to_view`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:45 in public method `allowed_to_vote`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:45 in public method `allowed_to_vote`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:48 in public method `anonymous_vote_id`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:48 in public method `anonymous_vote_id`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:52 in public class `Polls`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/polls.py:52 in public class `Polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:52 in public class `Polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:60 in public method `ct`:
        D102: Docstring missing
src/collective/polls/polls.py:64 in public method `mt`:
        D102: Docstring missing
src/collective/polls/polls.py:68 in public method `wt`:
        D102: Docstring missing
src/collective/polls/polls.py:72 in public method `member`:
        D102: Docstring missing
src/collective/polls/polls.py:75 in private method `_query_for_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:75 in private method `_query_for_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:81 in public method `uid_for_poll`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:81 in public method `uid_for_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:85 in public method `recent_polls`:
        D400: First line should end with '.', not 's'
src/collective/polls/polls.py:85 in public method `recent_polls`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:98 in public method `poll_by_uid`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:98 in public method `poll_by_uid`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:109 in public method `voted_in_a_poll`:
        D400: First line should end with '.', not 'd'
src/collective/polls/polls.py:109 in public method `voted_in_a_poll`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:128 in public method `allowed_to_edit`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D401: First line should be imperative: 'I', not 'Is'
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D400: First line should end with '.', not 'l'
src/collective/polls/polls.py:133 in public method `allowed_to_view`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D401: First line should be imperative: 'i', not 'is'
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D400: First line should end with '.', not '?'
src/collective/polls/polls.py:137 in public method `allowed_to_vote`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/polls.py:156 in public class `PollPortletRender`:
        D400: First line should end with '.', not 'w'
src/collective/polls/polls.py:181 in public method `render_portlet`:
        D202: No blank lines allowed *after* method docstring, found 1
src/collective/polls/polls.py:251 in public method `render`:
        D400: First line should end with '.', not 'e'
src/collective/polls/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/subscribers.py:1 at module level:
        D100: Docstring missing
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D400: First line should end with '.', not 'f'
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D208: Docstring is over-indented
src/collective/polls/subscribers.py:17 in public function `fix_permissions`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D400: First line should end with '.', not 't'
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D208: Docstring is over-indented
src/collective/polls/subscribers.py:37 in public function `remove_votes`:
        D300: Expected """-quotes, got '''-quotes
src/collective/polls/setuphandlers.py:1 at module level:
        D100: Docstring missing
src/collective/polls/setuphandlers.py:12 in public class `HiddenProfiles`:
        D101: Docstring missing
src/collective/polls/setuphandlers.py:18 in public method `getNonInstallableProfiles`:
        D102: Docstring missing
src/collective/polls/setuphandlers.py:23 in public function `updateWorkflowDefinitions`:
        D103: Docstring missing
src/collective/polls/setuphandlers.py:29 in public function `setupVarious`:
        D103: Docstring missing
src/collective/polls/portlet/voteportlet.py:1 at module level:
        D100: Docstring missing
src/collective/polls/portlet/voteportlet.py:24 in public function `PossiblePolls`:
        D103: Docstring missing
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D400: First line should end with '.', not 't'
src/collective/polls/portlet/voteportlet.py:46 in public class `IVotePortlet`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/portlet/voteportlet.py:87 in public class `Assignment`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:102 in public method `__init__`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D401: First line should be imperative: 'Thi', not 'This'
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D400: First line should end with '.', not 'e'
src/collective/polls/portlet/voteportlet.py:111 in public method `title`:
        D205: Blank line missing between one-line summary and description
src/collective/polls/portlet/voteportlet.py:118 in public class `Renderer`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:129 in public method `utility`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/portlet/voteportlet.py:135 in public method `portlet_manager_name`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:149 in public method `poll`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:170 in public method `poll_uid`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/portlet/voteportlet.py:176 in public method `getVotingResults`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:184 in public method `can_vote`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:193 in public method `available`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:202 in public method `is_closed`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:208 in public class `AddForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:208 in public class `AddForm`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:217 in public method `create`:
        D102: Docstring missing
src/collective/polls/portlet/voteportlet.py:221 in public class `EditForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/portlet/voteportlet.py:221 in public class `EditForm`:
        D204: Expected 1 blank line *after* class docstring, found 0
src/collective/polls/portlet/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/Extensions/Install.py:1 at module level:
        D100: Docstring missing
src/collective/polls/Extensions/Install.py:7 in public function `uninstall`:
        D103: Docstring missing
src/collective/polls/Extensions/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/tests/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/__init__.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/poll.py:1 at module level:
        D100: Docstring missing
src/collective/polls/content/poll.py:34 in public class `InsuficientOptions`:
        D101: Docstring missing
src/collective/polls/content/poll.py:38 in public class `IPoll`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:38 in public class `IPoll`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:80 in public method `validate_options`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D400: First line should end with '.', not 'e'
src/collective/polls/content/poll.py:90 in public class `Poll`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:101 in public method `annotations`:
        D102: Docstring missing
src/collective/polls/content/poll.py:105 in public method `utility`:
        D102: Docstring missing
src/collective/polls/content/poll.py:109 in public method `getOptions`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:115 in private method `_getVotes`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:133 in public method `getResults`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:144 in private method `_validateVote`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:155 in private method `_setVoter`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:180 in public method `voters`:
        D102: Docstring missing
src/collective/polls/content/poll.py:186 in public method `total_votes`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:192 in public method `setVote`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:217 in public class `PollAddForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:217 in public class `PollAddForm`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:223 in public method `create`:
        D102: Docstring missing
src/collective/polls/content/poll.py:235 in public class `PollEditForm`:
        D203: Expected 1 blank line *before* class docstring, found 0
src/collective/polls/content/poll.py:235 in public class `PollEditForm`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:241 in public method `updateWidgets`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:258 in public method `applyChanges`:
        D102: Docstring missing
src/collective/polls/content/poll.py:270 in public class `View`:
        D101: Docstring missing
src/collective/polls/content/poll.py:277 in public method `update`:
        D102: Docstring missing
src/collective/polls/content/poll.py:346 in public method `can_vote`:
        D102: Docstring missing
src/collective/polls/content/poll.py:357 in public method `can_edit`:
        D102: Docstring missing
src/collective/polls/content/poll.py:362 in public method `has_voted`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:371 in public method `poll_uid`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:377 in public method `getOptions`:
        D200: One-line docstring should not occupy 2 lines
src/collective/polls/content/poll.py:382 in public method `getResults`:
        D200: One-line docstring should not occupy 2 lines

Wednesday, January 8, 2014

How to get statistics about your contributions on a GitHub organization

GitHub, the web-based hosting service for software development projects, lets you check statistics on your contributions on specific projects easily.

You can check, for instance, the statistics on collective.cover just by visiting the following page:

https://github.com/collective/collective.cover/graphs/contributors

GitHub gives you a count of commits and a visual representation of them among time.

If you want to check only the number of commits by author, you could use the git-shortlog command:

The -s option gives you a summary output and the sort command shows the information sorted in reverse order (-nr).

You could improve this listing by using the .mailmap feature to add commits belonging to the same author using two different email addresses.

Image now that you want to get a list of all your contributions on a specific organization, lets say, the Plone collective.

Enter the GitHub API and github3.py, a Python wrapper for it.

Until today, the Plone collective has more that 1.1k repositories so we need to use the authenticated access to the API to avoid depletion of requests (rate limit allow us to make up to 60 requests per hour for unauthenticated requests and 5,000 for authenticated requests). In this specific example we used 1,171 requests to the API.


Using github3.py we iterate over all of the organization repositories and get information of each one only if my user name is listed among the contributors. We get the results as a list of tuples (repo, total).

After that we sort the list in reverse order using the total commits as the key and make the sum of all commits in general.

As you can see I have contributed to 74 repositories on the Plone collective making 2,975 commits in total.

Not bad, isn't it? :-)

Wednesday, August 8, 2012

Integrating Travis CI with your Plone add-ons hosted on GitHub

Update 15/9/2012: Kudos for Asko Soukka who has developed an alternative method of installing Plone using the old good universal installer that reduces the amount of time needed by half. So go and read his post instead of loosing your time with mine.

I took me a little bit but, with the help of Mikko and Martin, I've got a couple of add-ons running tests with Travis CI.

Before setting up Travis CI, you have to make some changes to the Setup Script of your package.

In my case, my add-on package only works for Plone versions 4.1 and later, so I have added Products.CMFPlone as a dependency:

    …
    install_requires=[
        'setuptools',
        'Products.CMFPlone>=4.1',
        ],
    extras_require={
        'test': ['plone.app.testing'],
        },
    …

Products.CMFPlone contains a cut down feature set: just the things I need to run my tests.

Setting up Travis CI is pretty easy: just sign in and activate your GitHub Service Hook

Now, you need to configure your Travis CI build with a .travis.yml file in the root of your repo:

In my case I'm running tests for Plone's latest stable release (4.2 at I write this post) on top of Python 2.7.

Let's take a look at the travis.cfg buildout configuration file:

The main issue I experimented on my first tests was timeouts, so I have a couple of tricks here for you: first, we are extending the standard Plone testing buildout configuration that includes most declarations for us and takes care of always running the latest stable version; we are only going to use the test part. You need to add the package-extras just if your add-on is using plone.app.testing on the test option in extras_require of your package declaration as mentioned above.

zope.globalrequest is needed to run the tests and it was not included on Products.CMFPlone (this is already fixed on Plone's branches for versions 4.2 and 4.3). You may also need to include Pillow in test-eggs; just uncomment it the line.

We also need to add a socket-timeout of 3 seconds (only available on zc.buildout >= 1.5.0) and a list of allow-hosts to download the dependencies. This is pretty important and will avoid timeouts as Travis CI has hard time limits and timeouts are between 10 and 15 minutes for test suite runs (1).

Last, we have to replace the eggs option on the test part; we need to do this because we don't want to include neither Plone or plone.app.upgrade on the tests.

Finally, you can add a Status Image with a link back to the result of your last build on your README.txt file:

.. image:: https://secure.travis-ci.org/collective/your.package.png
    :target: http://travis-ci.org/collective/your.package


To run the tests you only need to make a push to your GitHub repo. Easy, isn't it?

For a live example of all I mentioned above, take a look at the collective.prettydate package.

Travis CI is really easy to set up and fun to use; I strongly recommend it and, if you like it, please show your love donating.

Thursday, June 7, 2012

Ray Bradbury in memoriam

The Illustrated Man turned in the moonlight. He turned again… and again… and again…
Ray Bradbury (The Illustrated Man, 1951)


Ray Bradbury, author of Fahrenheit 451 and one of the greatest science fiction writers of the 20th century, died at 91 two days ago.

I leave with you, as an homage, a fragment of one the most beautiful stories written by him.

Kaleidoscope (fragment)

[…]

The many good-bys. The short farewells. And now the great loose brain was disintegrating. The components of the brain which had worked so beautifully and efficiently in the skull case of the rocket ship firing through space were dying one by one; the meaning of their life together was falling apart. And as a body dies when the brain ceases functioning, so the spirit of the ship and their long time together and what they meant to one another was dying. Applegate was now no more than a finger blown from the parent body, no longer to be despised and worked against. The brain was exploded, and the senseless, useless fragments of it were far scattered. The voices faded and now all of space was silent. Hollis was alone, falling.

They were all alone. Their voices had died like echoes of the words of God spoken and vibrating in the starred deep. There went the captain to the Moon; there Stone with the meteor swarm; there Stimson; there Applegate toward Pluto; there Smith and Turner and Underwood and all the rest, the shards of the kaleidoscope that had formed a thinking pattern for so long, hurled apart.

And I? thought Hollis. What can I do? Is there anything I can do now to make up for a terrible and empty life? If only I could do one good thing to make up for the meanness I collected all these years and didn’t even know was in me! But there’s no one here but myself, and how can you do good all alone? You can’t. Tomorrow night I’ll hit Earth s atmosphere.

I’ll burn, he thought, and be scattered in ashes all over the continental lands. I’ll be put to use. Just a little bit, but ashes are ashes and they’ll add to the land.

He fell swiftly, like a bullet, like a pebble, like an iron weight, objective, objective all of the time now, not sad or happy or anything, but only wishing he could do a good thing now that everything was gone, a good thing for just himself to know about.

When I hit the atmosphere, I’ll burn like a meteor.

“I wonder,” he said, “if anyone’ll see me?”



The small boy on the country road looked up and screamed. “Look, Mom, look! A falling star!”

The blazing white star fell down the sky of dusk in Illinois. “Make a wish,” said his mother. “Make a wish.”


(1949)

Photo: Eneas.

Tuesday, June 5, 2012

Running pep8 before any commit on Git

Readability counts.

One of the things that made me choose Python as a programming language in the first place was its readability.

PEP 8 —the Style Guide for Python Code— gives coding conventions for the Python code and pep8 is a tool to check your code against these conventions.

I use gedit as my editor and I have installed the developer plugins to check my code against PEP 8 every time I save a file but, as not everyone does this, Érico asked me today about a way to enforce this practice.

I started searching the web and I have compiled (from 1, 2 and 3) a nice solution using a pre-commit hook with Git (this is possible also with Subversion, but I'm not pretty interested on it right now):

First you need to be sure you are running Git version 1.7.1 or later, and that you have pep8 installed in your system (check the package documentation).

Create a directory to store the hooks globally:

mkdir -p ~/.git_template/hooks

Tell Git all new repositories you create or clone will use this directory for templates:

git config --global init.templatedir '~/.git_template'


Put the following script in the ~/.git_template/hooks directory:


Make the file executable:

chmod +x ~/.git_template/hooks/pre-commit

If you want to use this hook on an existing repository all you have to do is reinitialize it:

git init

Now the pre-commit hook script lives in the .git/hooks directory of your repository.

Test it trying to commit some changes: if the files you are trying to commit comply with PEP 8 (excepting the list of errors or warnings to ignore), the commit will be done as usual; if there are any issues, the commit will be aborted until you fix them.

Feel free to modify the list of errors and warnings to ignore, globally or from project to project, to fit your personal needs.

Remember PEP 8:

A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is most important.

But most importantly: know when to be inconsistent -- sometimes the style guide just doesn't apply. When in doubt, use your best judgment. Look at other examples and decide what looks best. And don't hesitate to ask!

Two good reasons to break a particular rule:
  1. When applying the rule would make the code less readable, even for someone who is used to reading code that follows the rules.
  2. To be consistent with surrounding code that also breaks it (maybe for historic reasons) although this is also an opportunity to clean up someone else's mess (in true XP style).