I've fiddled with my blog template because I decided I wanted more horizontal viewing space, given that it was using less than a third of my 1920 horizontal pixels. If it feels too spread out for you, I added a drag-and-drop handle over to the left to let you resize the main content column. The javascript is pretty primitive. If it breaks, drop me a comment.

Wednesday, July 8, 2009

How To Aggregate Downstream Test Results in Hudson

I must've googled a dozen permutations of this post's title looking for the key to my problem of... well, what it says. How on earth do I get Hudson (a top-notch CI server) to "Aggregate Downstream Test Results"??? The little help icon, which is typically very useful, provided a handy description of the feature, but didn't say anything about how to use it. My searches found several posts from other people having the same problem, but little in the way of resolution. I'm not sure if there's just not many people wanting to do this or if the "how" was just very obvious to everyone but me, but one way or the other, I finally figured it out. In retrospect, it does seem a bit obvious.
The key to getting Hudson to track test results from downstream builds is that all of the builds, upstream and down, must have a common, fingerprinted artifact. Any old artifact will do...almost. More on that later. I just had my first job--let's call it Foo--create a file with the date and time in it called "starttime".
1) Have Foo archive this file as an artifact, and either enter the file in the list of artifacts to fingerprint, or turn on fingerprinting for all artifacts.
2) In the downstream job, Bar, do exactly the same as in step one for Foo: archive and fingerprint the same file. Of course this implies that Bar has actually gotten the file in question somehow. I haven't worked out how to do this through Hudson other than by using wget to retrieve the archived one from the last build of Foo: something like "wget http://localhost:8080/job/Foo/lastBuild/artifact/starttime".
3) Of course, you'll need to have some test output being recorded by at least Bar, if not both jobs, since the whole point of this is to see aggregated test results.
4) You can always see the very latest test results at http://localhost:8080/job/Foo/lastSuccessfulBuild/aggregatedTestReport. Don't use "lastBuild" or the page will be unavailable when Foo is running or broken. When you have a lot of downstream tests, you can enable auto-refresh on this page and see the results fill in as jobs complete--very cool, especially when you have a nice test cluster with lots of downstream jobs.
5) It's handy to make Bar triggered by a build of Foo, but it's by no means required. From now on, any builds of Foo and Bar that share an identical artifact (checked by md5 checksum) are linked together, and the link appears in several places, one of which is the aggregated test results.
I emphasized the word "any" there for the same reason I said you can use almost any artifact. Remember it's the md5sum of the artifact in question that is used to determine which builds of which jobs are linked together. Say you start builds Foo #4, #5, and #6, and all create or use the same "starttime" file for some reason--maybe you're using just the date instead of date + time. Then Bar #4, #5, and #6 also all retrieve, archive, and fingerprint the same file. Since the checksum of the artifact is common in all those builds, they'll all be linked together, and Hudson can't really tell them apart. I'm not sure what this does for aggregated test results, but in other places, like where upstream and downstream builds are listed, instead of showing that Foo #4 led to Bar #4, it'll show Foo #4 led to Bar #4-#6.
So there ya go. Let 'er rip. I'm happily chugging away now with a 5-node Hudson cluster churning out test results like there's no tomorrow.


Jonathan said...

You can use the URL SCM plugin in Hudson to retrieve the contents of the lastSuccessBuild rather than using wget.

Ryan said...

Thanks for the tip, Jonathan! That looks like exactly what I need to fill that gap. I'm having trouble getting to the Hudson site right now, but I'll definitely check it out later.

mdonohue said...

The wiki page for URL SCM could use some snazzing up, but it is here

├ůsmund said...

I had somehow been able to understand most of what you have written.

But I did find one other funny issue. In my setup the aggregation of test results did not happen if fingerprint did not change after enabling the test aggregation. ;-)

Workaround is to make a code change that change fingerprint.

Anonymous said...

Many thanks. Hudson is pretty easy to use most of the time, but this feature was rather confusing.

John said...

I was able to solve the passing of the artifact problem by having both builds build in the same directory. This can be done by selecting "Use Custom Workspace" under the "Advanced Project Options" and specifying the directory you want to use. You will have to do this for both projects, and you will probably want to also set the options to block building while up/downstream projects are building.

josesa said...

Use CopyArtifacts Plugin to get the artifacts archived in the upstream jobs.

Nicolas said...

I'm struggling with the same problem using the very latest Jenkins ver. 1.481. Thanks everyone for your tips, especially on how to use copyArtifacts. So I followed everything to the T. I use fingerprinted artifacts that I confirm are common between the upstream project build and the downstream builds. My artifact is unique to every new build of the upstream project as well. But alas, I always get "Aggregate Test Result (no tests)". Whats the deal ?!? I do not publish Junit tst result reports nor do I publish javaDoc, but those aren't required right ? Please help !

Zachary Young said...

@Nicolas: I was in your shoes about an hour ago: you do need to publish Test Results for the Aggregator to work. Specifically, you need to publish JUnit Test Results as a post-build action. I'm actually using Canoo Web Tester, and using the WebTest Presenter post-build action was not enough: I needed both. JUnit to aggregate upstream, and WebTest Presenter to actually show the errors.

Zhanglei Wang said...

I did everything mentioned in the post and it just does NOT show any aggregated test results...

The only difference is that I use the "Groovy postbuild" plugin to launch the downstream jobs, so the "downstream builds" link does not work at all. But the "fingerprints" link does work. I can see the related upstream/downstream jobs.

I can see the test results from each downstream job. But the "aggregate" link from the upstream job just does not work!

Ryan said...

I'm sorry I can't help you guys with new issues. This post is from a version of Hudson about 4 years ago, and while I still use Jenkins, I'm no longer running a distributed build and aggregating test results.

Zachary Young said...

I just used this post as a springboard to get my own Jenkins jobs to aggregate. I documented my steps in the form of this SO answer for an updated Jenkins approach: