In Valid Logic

When configuring our Selenium tests to run out build server, one issue I ran into is that there wasn’t a lot out there in terms of how to get this set up. A few people had posted about having done it, but not how they did it. Overall, it isn’t incredibly difficult, but there are a few things that I got hung up on.

The way we run it is we start the selenium server in the background, and then have NUnit tests that run the stories. We can’t start Selenium in the foreground because then we couldn’t move on to the NUnit tests, but we also needed to register it as a Windows service on demand. To do that, we used a utility called Non-Sucking Service Manager (NSSM). It is like srvany from the Windows Resource Kit, but it works on Windows Server 2008 (and there was no publicly available Windows Resource Kit for 2008 I could find).

Here is our setup/teardown NAnt targets:

<target name="start.selenium">
	<property name="jdk.bin" value="${environment::get-variable('JDK_BIN')}" />
	<exec basedir="${common.folder}" program="nssm.exe" commandline='install Selenium "${jdk.bin}\java.exe" "-jar ${source.folder}\libs\selenium-server.jar"' />
  <exec program="net.exe" commandline="start Selenium" />
</target>

<target name="stop.selenium">
  <exec program="net.exe" commandline="stop Selenium" failonerror="false" />
	<exec basedir="${common.folder}" program="nssm.exe" commandline="remove Selenium confirm" failonerror="false" />
</target>

On top of that, it is important to set the “nant.onfailure” property to a target that does all of your cleanup. Since we’re using NUnit, an error in the unit tests will signal a build failure, but we still want to ensure our cleanup runs. So set the “nant.onfailure” property to the name of a target that runs all cleanup tasks (including “stop.selenium”).

Proper cleanup is very important, since in our case, the selenium.jar is in with the source and if it is still running, Team City will be unable to cleanup the working space and cause it to instantly fail any future builds before it even gets to executing your script. Because of this, it might also be good to have the server do a scheduled task that calls “net stop Selenium”. If it is stopped already, no harm, if the service doesn’t exist, no harm, but just a bit of an extra measure. And finally, to be extra extra safe, I actually call our main cleanup target at the beginning of the build as well and set failonerror=”false”. Since we’re registering and running background processes, didn’t want the test to be brittle and need my intervention. I don’t want “Ken, can you fix the build” emails.

So few things to point out about using it:

• One thing I got hung up on, you need the quotes around the “-jar …” parameter on the exec line to nssm.exe. I was missing that originally and wasted a lot of time, since I’d see the Java process start and be running in the process list, but couldn’t communicate with the Selenium server. The “-jar …” line is a the parameter to java.exe though, and not nssm.exe, so it needs to know it is all one parameter.
• We have the location of the JDK set in the environment variables. I don’t like hardcoded paths in the scripts.
• Be careful of quotes! I had to have the JDK_BIN set to a property because otherwise I’d get down to invalid quotes. You can’t have commandline=’….”-jar ${environment::get-variable(‘JDK_BIN’)}”‘…. the second set of single-quotes will be closing the first one, not opening a new set.
• The “common.folder” and “source.folder” are just properties pointing to where some common utilities are and the current working directory for the source.

Recently, I’ve been working on migrating our build process at Telligent from using Cruise Control .NET to use JetBrain’s Team City. Cruise Control has served us well, but after looking at the feature set of Team City, it had a number of things that we were wanting to leverage as we continued to expand our build process. Was asked on Twitter what I liked so much about Team City versus Cruise Control, but there wasn’t a good way to sum it up in 140 characters. So here is a brief list of some of the reasons:

1. No XML configuration files. Granted, our builds are in NAnt, so we’re still married to XML, but creating a new build doesn’t need to take any XML editing anymore. Our build scripts have no hard coded values, everything it needs it passed in from Team City. So we can go an existing build, click Copy, then just edit the configuration settings as needed.
2. It queues builds. We run a number of daily builds at 4pm Central and at that time, the build server comes to a screeching halt because all of them run at the same time. We’ve had to stagger them an extend the timeout limit on several because of how slow it gets. Team City instead runs one at a time, queues the others, and lets you prioritize the queue if you need to bump things up.
3. Does more for you. First of all, it exports the source from the main repository cache for you. When building, you want to avoid working off the main repository cache since then you can get stuck with collisions on update, but with CC.NET, we always had to do the export and everything ourself. Team City takes care of and reverts the source to a clean state one each run.
4. Artifacts and cross dependencies. In CC.NET, we always had a network share set up where we’d place built packages, but with Team City, they can be loaded back into the main site for storage and easily accessible. On top of that, you can place dependencies between builds and have the artifacts extracted. For instance, we have one built that fully packages a release every day. We have another build that our Selenium tests. We have a build dependency on them so that Team City extracts our latest package and the build script sets it up in IIS and runs against it. As time goes on and we get more of our subcomponents into Team City, ensuring all subcomponents stay up to date will be automated and far easier.
5. Streaming build log. No longer do I need to wait for the build to be done to see what is going on. Team City can stream the build log to the browser, so I can see where its at, what is hung up on, or if it did something wrong that wasn’t an error, can stop the build and correct it.
6. Easily customize builds. A while ago, we’d built a “preview build” and then a short while afterwards, realized there was an issue with the build. Not with the code, but just the build process. We needed to send an updated one, but wasn’t sure if the latest code was at a release-worthy state. Luckily, Team City makes it super easy. Right through the interface, I was able to kick off a new build of the same SVN revision as the earlier one. It also was intelligent enough so that in the history, it distinguishes it so someone knows it is different and in the description, says it is a build of an earlier commit and out of line with the other builds.
7. Test history. We’ve just started pulling in our test builds, but one nice thing I’ve immediately notice is some of the rich reporting it has around test stats. For instance, it shows when a test started failing (if its been in multiple builds), allows you assign someone to be responsible, allows you to track a test’s history, and even identify problematic tests (probably ones that fail the most). You can even track the test duration and see it change over time.

Overall, would have to sum Team City up as just being incredibly refined.

I know a number of people who have larger Cruise Control setups than us, but overall Team City brings a lot to the table and the cost factor makes it a dead on winner. Team City can easily pay for itself just in terms of productivity. But on top of that, it is free up to 20 build configurations!