rapaul.com

From zero to headless browser tests in Jenkins

After spending a large portion of the day I can proudly say I have a working set of browser based tests that run on a headless Jenkins install. By headless I mean a server without any physical display installed, as is typical for server machines. This facilitates the execution of high level acceptance tests in much the same fashion as lower level unit and integration based tests, albeit at a slower rate.

The problem I am trying to solve here is a quick feedback loop on acceptance test level behaviour. This blog post will be talking about getting Cucumber scenarios running for a single browser (Firefox). Cross browser testing is a different problem, for which think Sauce Labs would be a better solution as they take the hassle out of provisioning and maintaining a wide range of operating system and browser combinations.

Outlined below are the steps I followed to go from installing Ubuntu server edition through to running the browser based tests (with Cucumber, Capybara, Selenium-Webdriver). You may find some steps are not required on your operating system or for the project you wish to test. Admittedly I dove down a few rabbit holes, but thanks to VirtualBox’s snapshot feature I could safely revert if things turned sour.

• Installing Ubuntu
• Installing Jenkins
• Going Headless
• Installing Ruby with RVM
• Installing Firefox
• Creating a Job
• Bonus Points, watching the browser in realtime

Installing Ubuntu

If you have an existing server you can skip this step. If not grab yourself a copy of Ubuntu Server edition. As I wanted a simple way to play with Jenkins without provisioning hardware I used VirtualBox for virtualisation. I followed the usual VirtualBox installation, however once installed the server showed a blank screen on boot. This was fixed by following the workaround on the Ubuntu forums.

When installing, it is handy to enable the OpenSSH server so you can SSH onto the box from your desktop terminal, this makes copy & pasting some of the later steps much easier. To make the VirutalBox server visible on the network, change the network mode from NAT to bridged.

Installing Jenkins

Installing Jenkins is a breeze, as debian packages have been set up, check the wiki page for details.

This automatically creates an account called jenkins. We will need to login as this user later so set a password for jenkins with:

You should now be able to view the Jenkins dashboard at http://your.server:8080/

Going Headless

Now that Jenkins is installed, we want to get a headless display configured for our browser based tests. First up hit Manage Jenkins > Manage Plugins > Available and install the Hudson Xvnc plugin (this works with Jenkins despite its name). Schedule Jenkins to restart to pickup the plugin. Once installed this gives us the ability to start a headless display automatically when we configure our jobs, more on that later.

With Jenkins configured we need to ensure the required software is installed on the server:

vncserver requires a password to be set before it can be used, this needs to be set before Jenkins can make use of the vncserver. For this we need to switch to the jenkins user and set a password.

When Jenkins runs it doesn’t need to know this password, but if you want to watch a running job you can connect to the running vnc session with that password and watch the tests in real time.

I initially headed down the Xvfb route but that seemed to require a lot of custom configuration in the job’s build script and isn’t related to the Xvnc plugin.

Installing Ruby with RVM

The job I’m wanting to run is a set of acceptance tests written in Cucumber with automation done using Capybara (Selenium-Webdriver under the hood). So its a Ruby job, and all good Ruby jobs use RVM. Fortunately RVM has a page on integrating with Hudson/Jenkins. I followed the recommended steps and installed RVM for a single user (jenkins).

Once RVM is configured, run rvm notes to find the full list of dependencies you need to install for your required version of Ruby. e.g.

Note that I didn’t give the jenkins user sudo rights, so I installed all packages through my usual admin account on the server.

RVM can be configured to allow the automatic installation of Ruby versions and gemsets by adding the following to ~/.rvmrc for the jenkins user:

Installing Firefox

Of course, a headless server isn’t any good without a browser to test

This is the default browser that selenium will select.

Creating a Job

At this point the Jenkins server should be fully configured to run headless jobs, so lets dive in and create one. Create a new freestyle job. Notice there is a new option available under the ‘Build Environment’ section call ‘Run Xvnc during build’, check this to have the plugin automatically do its magic.

For my example, I didn’t bother with checking projects out source control, I simply created a project in the /tmp directory. You’ll want to enable the appropriate SCM plugin and configure a checkout.
Under the build section add an Execute shell step with the following:

The -e flag in #!/bin/bash -e ensures the script stops after any errors.
You will notice that the script sources the .rvmrc file directly for the project, this ensures the correct version of Ruby is used with a gemset appropriate for your project. My .rvmrc looked something like:

Calling bundle install automatically installs bundler, reads the Gemfile.lock and installs all required gems. Finally cucumber kicks off the actual cucumber scenarios, and fingers crossed, they should pass with flying colours.

Bonus Points, watching the browser in realtime

As the headless display is running in vncserver, you can connect to the vnc session and watch the tests run in real time. Just use your regular VNC client and connect to your.server:59xx where xx is the display number output on the Jenkins console for the running job. You will need to enter the password you set the first time you ran vncserver.

[Note: most/all of these instructions should work with Hudson also]

Tags: capybara, ci, cucumber, jekins, rvm, selenium, virtualbox, vnc, webdriver

7 Comments to From zero to headless browser tests in Jenkins

Donald Piret
June 13, 2011

Hi,

Thanks for the great article. I’d be interested to know what combination of versions of capybara, cucumber and selenium-webdriver you got this to work with. I keep running into issues with different combinations of each lib.
Also what do you do about running your tests without transactions for selenium?

Richard Paul
June 13, 2011

Hi Donald, please you found the article useful.

capybara (0.4.1.2)
cucumber (0.10.0)
selenium-webdriver (0.2.0)

However in my Gemfile I simply have:

source ‘http://rubygems.org’
gem ‘cucumber’
gem ‘capybara’

So at the time I ran the bundle install it just selected those versions. If you aren’t using bundler I suggest you check it out.

As for transactions, the way our application works actions are confined to a single user so it isn’t a problem. But I know certain frameworks, such as rails, have hooks for rolling back transactions automatically.

Jeff Ancel
July 11, 2011

This is a wonderful writeup. Thanks for taking the time to help us all out.

Paul Caporn
August 2, 2011

Thanks for a great write up. Was struggling to get Xfvb and firefox to play and the vnc option bypassed it nicely

Alister Scott
August 29, 2011

You can use the ‘headless’ gem to achieve the same thing. See: http://watirmelon.com/2011/08/29/running-your-watir-webdriver-tests-in-the-cloud-for-free/
and http://jenkins.watirmelon.com/

Paresh Mahajan
January 31, 2012

hi,
In my case, Jenkins server is installed on windows server 2003 where I dont have vnc server installed.
So to view the headless execution of browser tests (in my case selenium), I have only one option to remote desktop that machine.

Now, though I connect via remote desktop to the server, I cant see the scripts running with opened browser. I have installed Xvnc plugin and checked the same in job configuration.

Can you help me here: how to see the opened browser to debug some failing script on jenkins server(which passes locally)?

Richard Paul
February 1, 2012

Hi Paresh,

I’ve not tried installing Jenkins on Windows, but I’m fairly sure Xvnc won’t work as it is related to http://en.wikipedia.org/wiki/Xorg which is generally used on Linux type systems.

The Jenkin’s mailing list might be your best port of call.

Leave a comment