For a long time I've thought that it would be a good idea to have a Continuous Integration server that could run all the tests for a project on checkin to source control. Until I started making use of Cucumber I didn't really have a full test run that took long enough to make buying a separate CI server feel more like an investment than an indulgence.
And then I started using Selenium to run in-browser tests for the Ajaxy parts of site, and Selenium is slow. Really slow.
Suddenly, a thirty-second full test run, which had crept up to one minute or so with Cucumber, was taking six minutes. What to do?
Enter CI, stage left.
If you've got more than one developer on a project then CI's a no-brainer. Knowing that your last commit and your colleague's last commit together break things, and knowing it as soon it happens, is priceless knowledge.
If you've got a large, layered, suite of tests, then it's also a fairly obvious one. For example, in a typical Rails app I tend to have three layers: Fast-running lower-level RSpec examples covering models, controllers, and views in isolation. Above that are end-to-end acceptance-level Cucumber scenarios which use Rails' integration testing (so, no browser involved), which take a lot longer to run. And above that are the Cucumber scenarios which drive a real browser for testing things (typically involving Ajax or something else Javascripty). Those take a long time to run.
Waiting for a long test run can really suck the productivity out of you. I don't generally need to run the Cucumber test layers as often as I do the RSpec layer (which I run constantly through Autotest). In fact, I can usually get away with only running all of them when I've done enough work to warrant a commit. (Obviously, if I'm working on something Ajaxy I'll run those specific features). So, why not let a magic box check out my commit and run all those tests for me. And, let's be honest, I'm not going to take that six-minute test run hit often enough if a machine doesn't do it for me.
I've looked at various bits of CI software over the years I've been skirting round this issue. Buildbot, Integrity, CruiseControl, CruiseControl.rb all got poked at to some extent, some I tried installing and configuring, I even succeeded with CC.rb, and others looked too complex to set up over lunch.
Buzz was building around Hudson, but I didn't get much past my knee-jerk anti-confluence response. Reader, I was a fool. After much prompting (Thanks, James Governor!) I installed, configured, and got a basic test run with Hudson over a lunchtime. Hudson wins.
I originally followed, broadly, this Hudson setup blog post over at juretta.com and evolved the setup so I could get test reporting for RSpec and Cucumber. Because the ground to cover for basic setup is so short, I'm going to cover everything from download to build again. I'm assuming you're on Mac OS X, but everything will work on *nix varieties unless I explicitly say otherwise. You can use Hudson with Windows, and there are instructions on their site.
The basic steps look like this:
Dead simple, this. Either download the latest version from this page, or from this direct link: http://hudson.gotdns.com/latest/hudson.war. (If you're on Debian or Ubuntu, there are packages) Once that's downloaded, run it from the command line like this:
java -jar /path/to/downloads/hudson.war
Visit http://localhost:8080/ in your browser. If you see the Hudson dashboard page, it works.
Hudson uses plugins to extend its core functionality, and you can install them through Hudson's own web interface. The whole process is painless, which was a nice surprise: I'd thought 'Java' and panicked slightly.
You can see available plugins in the 'available' pane of the plugin section in your Hudson management interface (http://localhost:8080/pluginManager/available). They're installed by checking the box next to their name and pressing the install button at the bottom right of the page. You can install several plugins at once, and once the process is over you need to restart Hudson.
You'll definitely need the Rake plugin. Hudson supports Subversion out of the box, but if you're using Git for source control you'll need a plugin.
A couple of quick notes. Unless gem dependencies are vendorised you'll need to ensure the box you're running Hudson has your dependencies installed.
spec, for example. You can add multiple tasks to the box. I tend to add more tasks to Hudson to ensure a clean environment for each run (this only caused me problems with the environment differences between the Selenium- and non-Selenium Cucumber runs).That's it for the basics. Hudson should be able to run your specs and will report a build fail if rake exits with a non-zero status code, which is exactly what you want. Hit the 'Build now' button in the sidebar to verify it all works. If it doesn't work, chances are that one of these two things has happened:
$PATH which Hudson is seeing. If rake and ruby work for you, then it's may be that you invoked Hudson from a different environment (like a system startup script) and that's the problem.There's lots of great intelligence you can get about your build from test reporting or metrics, but the number-one thing is to run the build after every commit. Fortunately, it's super simple. First, ensure that there's a public IP and port which points at Hudson's port on your box. Make sure you set up Hudson to require users to authorise - otherwise any old internet person could run arbitrary code on your box. See Hudson's guide to security for more information.
If you're just using the built-in server, it's really simple. When you start up Hudson, pass a username and password in as command-line args:
java -jar /usr/local/hudson/hudson.war --argumentsRealm.passwd.admin='adminpass' --argumentsRealm.roles.admin=admin
This creates a user. To enable security, click the 'Manage Hudson' link, select 'Enable Security', then check ''
Once you're secure, set up a post-commit hook in your source control system which makes an HTTP POST call to:
http://{your-IP}:{your-port}/job/{project-name}/build
That's it. Even if you've got username/password authentication set up, Hudson can allow anonymous POSTs to the project build URL with a simple auth token in the URL, precisely so that robots can start a build for you. Details are in the same Hudson security guide I mentioned before. I use GitHub's Service Hooks to start a new build after every git push.
Hudson can display and track test reports in JUnit's XML reporting format (It seems to be one of those great and mysterious undocumented formats that everyone relies on). Enabling it is as simple as checking the 'Publish JUnit test result report' option and specifying which files to report. The text box takes file globs relative to the Hudson workspace base dir, which is the root of the checked-out source (and so will probably be the same as RAILS_ROOT). I tend to use a glob which allows me to use different sub directories (one for each kind of run, e.g. RSpec, Cucumber-without-Selenium, Cucumber-with-Selenium), like this:
hudson/reports/**/*.xml
That'll catch any and all .xml files under hudson/reports. The next thing to do is get things to dump reports in the correct format.
Cucumber ships with a formatter for this -- the junit formatter. You set the formatter and point it at a directory, and away it goes. An example cucumber rake task for Hudson:
namespace :hudson do
def report_path
"hudson/reports/features/"
end
Cucumber::Rake::Task.new({'cucumber' => [:report_setup, 'db:prepare']}) do |t|
t.cucumber_opts = %{--profile default --format junit --out #{report_path}}
end
task :report_setup do
rm_rf report_path
mkdir_p report_path
end
end
The :report_setup task is worth a quick look. It ensures that your test report directory is clean, and not littered with files from a previous run, which could introduce misleading noise into your reports.
RSpec is a little bit trickier. There's no built-in JUnit formatter, for starters. There is the ci_reporter gem, however, which neatly rides to the rescue. ci_reporter provides a JUnit RSpec formatter. It's not entirely straightforward, because it needs to do some magic before the specs run, but it's pretty simple and it's easy enough to prevent it doing its magic unless you explicitly want it doing that. Sample tasks look like these:
namespace :hudson do
task :spec => ["hudson:setup:rspec", 'db:migrate', 'rake:spec']
namespace :setup do
task :pre_ci do
ENV["CI_REPORTS"] = 'hudson/reports/spec/'
gem 'ci_reporter'
require 'ci/reporter/rake/rspec'
end
task :rspec => [:pre_ci, "ci:setup:rspec"]
end
end
The ci:setup:rspec is ci_reporter's own task. The hudson:setup:rspec requires the hudson:setup:pre_ci task, in order to load up ci_reporter, and the ci:setup:rspec task can then perform its magic. All you need to ensure is that ci:setup:rspec runs before the main spec task. The rake:spec task is simply an explicit reference to the top-level spec task, otherwise rake would get confused with the hudson:spec task I defined. (It's a bit like using ::ClassName in Ruby to get ClassName and not Module::ClassName.)
Setting up Selenium-RC for use with Hudson (through Cucumber, I'm assuming) is simply a matter of ensuring that the bits required for Selenium are present on the box, and that the ports for Hudson (by default, 8080) and Selenium's RC server (by default, 4444) don't conflict. See my article on using Selenium and Cucumber together for more details: all you need to do differently is to use the JUnit formatter if you want reporting. My own setup tends to use separate (and separately cleaned) subdirs of hudson/reports/features to stop the Selenium Cucumber run clobbering the others.
You need to get Hudson to start up automatically. There are instructions for most platforms on the Hudson site. On a Mac, that means a launchd startup script. It's perfectly straightforward to set these up. The question is whether you want Hudson to start as a system daemon (at machine startup), or as a global or user-specific 'agent' (on login, either for all users or one specific user). If you're going to be running Selenium then you'll need an agent because a web browser needs to be started from a logged-in GUI user's environment - it can't be started by a system daemon directly. You can, of course, set your machine to automatically log a user in.
A sample startup script looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Program</key>
<string>/usr/local/hudson/launch-hudson</string>
<key>Label</key>
<string>org.hudson-ci.hudson</string>
<key>OnDemand</key>
<false/>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
(Save this as a file called org.hudson-ci.hudson.plist.) You'll notice that I didn't directly invoke hudson, but used a launch script. That's needed if you use a system like MacPorts in order that you can set the environment up as you need it. Without adding to the $PATH variable nothing launched by Hudson would be able to locate anything installed through MacPorts. That script looks like this:
#!/bin/sh
export PATH=/opt/local/bin:/opt/local/sbin:$PATH
java -jar /usr/local/hudson/hudson.war --argumentsRealm.passwd.admin='adminpass' --argumentsRealm.roles.admin=admin
It sets $PATH and starts Hudson with arguments which make it require a user named admin with password adminpass. (See the Hudson security guide for more details about that)
To install these, simply copy them to the appropriate places. For the launchd script this is either /Library/LaunchAgents (for all users), or ~/Library/LaunchAgents for a specific user. I put the hudson.war file and the launch-hudson script into /usr/local/hudson, with appropriate permissions (755 for launch-hudson and 644 for hudson.war) and ownership (chown root:wheel for both). That stops local users modifying them (probably what you want). It doesn't stop local users reading them, so your Hudson install would be vulnerable to someone who had already compromised your machine (they could read the Hudson username and password). Although, if someone's compromised your machine a compromised Hudson install is probably the least of your worries...
Once you've got the scripts in place, simply logging out and then in again (or restarting your machine) should make Hudson start without your intervention. If you have problems, try running the launch script from the command line, or looking at your logs. If you're having problems with the launchd .plist then you could try Peter Borg's Lingon, which is a GUI interface to create these files. It's not maintained anymore, but it should do the trick (even on Snow Leopard). You'll want the 2.1.1 version a .zip, not the 1.2 .dmg which it sticks in the big box at the top.