How To Aggregate Downstream Test Results in Hudson

I must've googled a dozen permutations of this post's title looking for the key to my problem of... well, what it says. How on earth do I get Hudson (a top-notch CI server) to "Aggregate Downstream Test Results"??? The little help icon, which is typically very useful, provided a handy description of the feature, but didn't say anything about how to use it. My searches found several posts from other people having the same problem, but little in the way of resolution. I'm not sure if there's just not many people wanting to do this or if the "how" was just very obvious to everyone but me, but one way or the other, I finally figured it out. In retrospect, it does seem a bit obvious.

The key to getting Hudson to track test results from downstream builds is that all of the builds, upstream and down, must have a common, fingerprinted artifact. Any old artifact will do...almost. More on that later. I just had my first job--let's call it Foo--create a file with the date and time in it called "starttime".

1) Have Foo archive this file as an artifact, and either enter the file in the list of artifacts to fingerprint, or turn on fingerprinting for all artifacts.

2) In the downstream job, Bar, do exactly the same as in step one for Foo: archive and fingerprint the same file. Of course this implies that Bar has actually gotten the file in question somehow. I haven't worked out how to do this through Hudson other than by using wget to retrieve the archived one from the last build of Foo: something like "wget http://localhost:8080/job/Foo/lastBuild/artifact/starttime".

3) Of course, you'll need to have some test output being recorded by at least Bar, if not both jobs, since the whole point of this is to see aggregated test results.

4) You can always see the very latest test results at http://localhost:8080/job/Foo/lastSuccessfulBuild/aggregatedTestReport. Don't use "lastBuild" or the page will be unavailable when Foo is running or broken. When you have a lot of downstream tests, you can enable auto-refresh on this page and see the results fill in as jobs complete--very cool, especially when you have a nice test cluster with lots of downstream jobs.

5) It's handy to make Bar triggered by a build of Foo, but it's by no means required. From now on, any builds of Foo and Bar that share an identical artifact (checked by md5 checksum) are linked together, and the link appears in several places, one of which is the aggregated test results.

I emphasized the word "any" there for the same reason I said you can use almost any artifact. Remember it's the md5sum of the artifact in question that is used to determine which builds of which jobs are linked together. Say you start builds Foo #4, #5, and #6, and all create or use the same "starttime" file for some reason--maybe you're using just the date instead of date + time. Then Bar #4, #5, and #6 also all retrieve, archive, and fingerprint the same file. Since the checksum of the artifact is common in all those builds, they'll all be linked together, and Hudson can't really tell them apart. I'm not sure what this does for aggregated test results, but in other places, like where upstream and downstream builds are listed, instead of showing that Foo #4 led to Bar #4, it'll show Foo #4 led to Bar #4-#6.

So there ya go. Let 'er rip. I'm happily chugging away now with a 5-node Hudson cluster churning out test results like there's no tomorrow.

Using Oracle's OCI Driver With JDBC

I just spent a morning learning how to use the Oracle OCI driver instead of the thin driver, and it's worth recording for future reference. I think it used to be more complicated, and you had to actually install some Oracle app or other, but as of version 10.1.2 or so, the process is somewhat simpler and just requires a couple of downloads. If you've always avoided using OCI to connect from Java or just never had a reason to do it before, here's what you need to do: 1) Download the Oracle "Instant Client", which lets you use OCI. I don't know exactly what these terms really mean in Oracle-speak. Just go to this download page, select your platform, then find your database version and download the "Instant Client Package - Basic" and "Instant Client Package - JDBC Supplement" packages. (You can use the Basic Lite package in place of the first one if you only need English/western i18n stuff.) You'll be prompted to log in to or register for OTN, which is a free registration. 2) Make sure you use the SAME DATABASE VERSION of the JDBC thin driver. If you don't already have this JAR, it comes in the Instant Client zip file (named ojdbc14.jar). You can also get it from this page if you really feel the need. It's important that the versions match, because the thin client uses native calls to access the OCI stuff, and the file names are hardcoded (I suspect) into the thin driver. So if you use version 10.x of the JDBC driver, but version 11 of the OCI driver, you'll get an error like "java.lang.UnsatisfiedLinkError: no ocijdbc10 in java.library.path".

3) Extract the two Instant Client downloads into the same directory somewhere.

4) Create your tnsnames.ora file somewhere (if you don't already have one). This file describes how to connect to one or more Oracle databases to any Oracle software on your computer, such as the Instant Client. It's widely documented on the web, like on this page. If you have a RAC and don't know much about this file (like me before this morning), then someone else probably set up the cluster. Get a tnsnames.ora entry from them and just paste it into a text file somewhere. I just put the tnsnames.ora in the same directory where I extracted the Instant Client.

5) Add/edit two environment variables. First, add the directory from 3) to your library path. In Windows, this means adding the directory to your PATH. For Linux:

export LD_LIBRARY_PATH=<directory>

Then add a TNS_ADMIN variable that points to the directory that contains tnsnames.ora (the same as the library directory if you put it with the rest of the extracted files).

6) Use JDBC to connect like normal with these settings (and a username + password, of course, if applicable):

driverClass=oracle.jdbc.driver.OracleDriver
connectionUrl=jdbc:oracle:oci:@<database alias>

where <database alias> is the alias specified in tnsnames.ora. It's the left hand side of the initial assignment in a TNS entry--"FITZGERALD.SALES" in this example:

FITZGERALD.SALES =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = server1)(PORT = 1521))
    (CONNECT_DATA =
      (SID = ORCL)
    )
  )

That's it! Section 7.4.3 of the Oracle JDBC Developer's Guide has a good rundown of different ways you can use JDBC to connect with OCI once you get it installed. I don't claim that this is the best way to do any of this. I just know it works.

Put Your Screen Saver to Work

Ever intend to start your computer working on a big job when you walk away so it'll be done when you come back an hour later? Ever walk away and forget to start it? That was happening to me a lot, so I wrote a screen saver that runs stuff while it's active, thereby automatically making use of my computer's idle time. I'm posting it here in case any of you would like to use it.

Since there doesn't seem to be an attachment mechanism, it's embedded in the image to the left. Just follow the instructions on the image. I got this simple method of embedding the file in the image from here: http://www.tricksystem.com/2008/12/secretly-hide-any-file-inside-jpg-image.html. You should get a file out of it named "Jobs@Home.scr". That's the screen saver. Just put it in c:\windows\system32, and it'll be selectable as a screen saver. MD5 checksum if you want to check it:

$ cat Jobs@Home.scr.md5
1f190b9e2db877fe07d683f551d621a3 *Jobs@Home.scr

How it works:

First, it's Windows-only. Sorry Linux guys. I wrote and am using it on Windows XP Home and Professional. Next, it's quite primitive. There is no configuration of the screen saver itself. It's hard-coded to read from a file named "c:\ssavejobs". This should be a plain text file with a command per line. The screen saver reads the lines from the file and executes each one in sequence (not parallel). When it finishes executing one line, it removes it from the file and begins the next one, so the file acts as a FIFO queue. Each line is executed by running the following command in a new process:

c:\windows\system32\cmd.exe /c <line from file>

Note that the ssavejobs file is *not* a .bat file. It is simply a series of one-line commands, each of which is executed as an independent process.

Example ssavejobs:

copy c:\test.txt c:\test2.txt
del c:\test.txt
move c:\test2.txt c:\test.txt
c:\foo\bar\rip.bat d: c:\rippedMusic
c:\foo\bar\encode.bat c:\rippedMusic\*

With the above example, the screen saver, when it comes on, would copy a file, delete a file, move the file back where it was (just trivial stuff to give you an idea of how it works), then rip music from your d: drive to a directory on your c: drive, and finally encode all the ripped music (assuming you have a rip.bat and encode.bat file that will do that for you). Keep in mind that unless you expect to be away from your computer for long enough for all the jobs to execute, you should make each job as short as possible. That way, when you do come back, either the current job will finish quickly, or you can cancel the current job without losing a ton of work. I.e in the above example, it would be better to rip and encode each track (or whatever) individually than doing two, large batch jobs.

Caveats:

The big one is that the screen saver can only update the ssavejobs file when it's on. That means if you come back to your computer and kill the saver while it's in the middle of a job, the job will continue (it's a separate process), but that same job will also remain at the top of the ssavejobs file. If this happens, you should either kill the process and let it restart later or let the job finish and manually remove the top line from ssavejobs. I might try to fix this if it becomes too big a headache.

Another problem is that the screen saver starts its background work when it displays in the "Display Properties" screen saver tab--you know, the tiny preview of it that you see in the picture of the monitor. So you might inadvertently start it by going to the screen saver dialog. It also starts, obviously, when you use the Preview button.

A potential deficiency is that the stdout of the processes being executed isn't captured anywhere, so you have no logs of what happened. You can only look at whatever outputs you were expecting to see if thing went as expected. I'll probably change this to print stdout/stderr to a log file if I can figure out how.

Finally, the dots (the screen saver itself) are not my doing. They were in the skeleton code that I downloaded to explain how to write a screen saver. They're more exciting than a blank screen, so I've left them on for now.

Generic Non-Erasure?

I was coding away the other day and got this error:

Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String

on code that looks in essence like this:

public class Example {
    public static void main(String[] args) {
        List<String> stringList = legacyGetList();
        System.out.println("First element: " + stringList.get(0));
    }

    static List legacyGetList() {
        List untypedList = new ArrayList();
        untypedList.add(new Integer(5));
        return untypedList;
    }
}

It happens on the fourth line, the System.out.println. What's the deal? It can't be related to the type parameter, can it? Generic types are erased at runtime, right? Well, it turns out that if you change List<String> to List<Object> or just List, then it works. Why?

Because of a compiler optimization. String concatenation compiles into StringBuilder invocations (at least it has since StringBuilder was introduced). StringBuilder has several overloaded append(...) methods--one for each primitive type, one for java.lang.String, and one for java.lang.Object--for convenience and, I guess, performance. So ask yourself: when is a method selected for execution from a list of overloads? If you answered "at compile-time", then you're right. If you answered the other way, prove it to yourself:

public class AnotherExample {
    public static void main(String[] args) {
        foo((String) null);
        foo((Object) null);
    }

    static void foo(String s) {
        System.out.println("It's a string");
    }

    static void foo(Object o) {
        System.out.println("It's an object");
    }
}

This fact answers the "Why?" from above. In the example, the compiler knows that the List is a List<String>, and therefore when the string concatenation is compiled into StringBuilder invocations, it compiles to StringBuilder.append(String) instead of StringBuilder.append(Object). Then at runtime, a java.lang.Integer is passed to the append(String) method, which obviously won't work. The proof is in the disassembled class code. Disassembled code from the example above:

0:   invokestatic    #16; //Method legacyGetList:()Ljava/util/List;
3:   astore_1
4:   getstatic       #20; //Field java/lang/System.out:Ljava/io/PrintStream;
7:   new     #26; //class java/lang/StringBuilder
10:  dup
11:  ldc     #28; //String First element:
13:  invokespecial   #30; //Method java/lang/StringBuilder."<init>":(Ljava/lang/String;)V
16:  aload_1
17:  iconst_0
18:  invokeinterface #33,  2; //InterfaceMethod java/util/List.get:(I)Ljava/lang/Object;
23:  checkcast       #39; //class java/lang/String
26:  invokevirtual   #41; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
29:  invokevirtual   #45; //Method java/lang/StringBuilder.toString:()Ljava/lang/String;
32:  invokevirtual   #49; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
35:  return

Note the "26: invoke StringBuilder.append(String)". That's the bad method call I mentioned. You'll notice the compiler throws a checkcast on the line before, which is where our ClassCastException comes from. If we change the List<String> to List<Object>, we get:

0:   invokestatic    #16; //Method legacyGetList:()Ljava/util/List;
3:   astore_1
4:   getstatic       #20; //Field java/lang/System.out:Ljava/io/PrintStream;
7:   new     #26; //class java/lang/StringBuilder
10:  dup
11:  ldc     #28; //String First element:
13:  invokespecial   #30; //Method java/lang/StringBuilder."<init>":(Ljava/lang/String;)V
16:  aload_1
17:  iconst_0
18:  invokeinterface #33,  2; //InterfaceMethod java/util/List.get:(I)Ljava/lang/Object;
23:  invokevirtual   #39; //Method java/lang/StringBuilder.append:(Ljava/lang/Object;)Ljava/lang/StringBuilder;
26:  invokevirtual   #43; //Method java/lang/StringBuilder.toString:()Ljava/lang/String;
29:  invokevirtual   #47; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
32:  return

The checkcast is gone, presumably because there's no reason to check if something is castable to Object, and it now correctly calls StringBuilder.append(Object).

Cookies

The web is full of "the best chocolate chip cookie recipe"s, but here is one from a NY Times article that would truly be a grievous loss to civilization if it were to pass out of knowledge. According to the accompanying article, this recipe was created with the input of a number of master bakers from New York City. I believe it. This recipe makes what could be the absolute best homemade chocolate chip cookies I've ever had. The recipe is online, but I'm replicating it here. Note if you don't read the article: refrigerating for 24-36 hours before cooking is a very important step. Also, I used regular semi-sweet chocolate chips, and they turned out fine. Maybe they'd be even better with the fancy chocolate?

Time: 45 minutes (for 1 6-cookie batch), plus at least 24 hours’ chilling

2 cups minus 2 tablespoons

(8 1/2 ounces) cake flour

1 2/3 cups (8 1/2 ounces) bread flour

1 1/4 teaspoons baking soda

1 1/2 teaspoons baking powder

1 1/2 teaspoons coarse salt

2 1/2 sticks (1 1/4 cups) unsalted butter

1 1/4 cups (10 ounces) light brown sugar

1 cup plus 2 tablespoons (8 ounces) granulated sugar

2 large eggs

2 teaspoons natural vanilla extract

1 1/4 pounds bittersweet chocolate disks or fèves, at least 60 percent cacao content (see note)

Sea salt.

1. Sift flours, baking soda, baking powder and salt into a bowl. Set aside.

2. Using a mixer fitted with paddle attachment, cream butter and sugars together until very light, about 5 minutes. Add eggs, one at a time, mixing well after each addition. Stir in the vanilla. Reduce speed to low, add dry ingredients and mix until just combined, 5 to 10 seconds. Drop chocolate pieces in and incorporate them without breaking them. Press plastic wrap against dough and refrigerate for 24 to 36 hours. Dough may be used in batches, and can be refrigerated for up to 72 hours.

3. When ready to bake, preheat oven to 350 degrees. Line a baking sheet with parchment paper or a nonstick baking mat. Set aside.

4. Scoop 6 3 1/2-ounce mounds of dough (the size of generous golf balls) onto baking sheet, making sure to turn horizontally any chocolate pieces that are poking up; it will make for a more attractive cookie. Sprinkle lightly with sea salt and bake until golden brown but still soft, 18 to 20 minutes. Transfer sheet to a wire rack for 10 minutes, then slip cookies onto another rack to cool a bit more. Repeat with remaining dough, or reserve dough, refrigerated, for baking remaining batches the next day. Eat warm, with a big napkin.

Yield: 1 1/2 dozen 5-inch cookies.

Note: Disks are sold at Jacques Torres Chocolate; Valrhona fèves, oval-shaped chocolate pieces, are at Whole Foods.

Good Design in 42 Seconds

Caution: not for the novice programmer! Ready? Go.

Find a method that's more complex than average. Use "extract method" refactoring to decompose a complex method into smaller pieces: private methods with self-documenting (doesn't-need-comments) names and correspondingly well-defined roles, like createNewThis, storeThat, and processIntoTheOther. Then convert each of the private methods into an interface/implementation pair. Apply this across a project, and you find yourself with pyramids of objects. The very lowest-level business and integration operations are encapsulated by small building blocks at the bottom. Mid-level objects are no larger than the lower, but assemble the lower-level objects into useful sequences for doing work. Higher level objects continue extending this pattern of maintaining small size but assembling functionality from lower-level chunks. Then unit testing any single object is as simple as mocking the few objects it depends on and testing that it invokes its dependencies in the right order and passes them the right parameters under whatever conditions it's in charge of knowing about. Changing behavior is equally easy because simply writing a new implementation of an object that's <100 lines is nothing. The old implementation stays around for reference, including its supporting tests, or it and its tests can be removed. No more changing the code and then having to trace through old tests to see what needs to be fixed there. Polymorphic open/closed.

Ubuntu Video Problems or How I Learned To Stop Worrying and Love Installing ATI Drivers

I've finally gotten around to fixing my broken Ubuntu desktop. It required a reinstallation of ATI video drivers for my Radeon X1900 card. The Situation: I had installed a fresh copy of 8.04 just after it came out, and I updated via the update manager as updates became available. A couple months ago, I installed some updates which upgraded me to kernel version 2.6.24-19-generic. I don't know what else changed, but after this I couldn't get to the desktop. I removed gdm from the init scripts so I could work at the command line. Every time I tried to run startx, though, it would first go to the grey/black and white check background with the "X" mouse cursor, then to a plain white screen with the normal Gnome mouse cursor. The mouse was responsive, but there was only the white screen. Hitting Alt+Tab would show the outline of what appeared to be a dialog box with the outline of the "task switch" panel in front of it. Hitting enter would make the dialog go away so that this didn't happen anymore and I was just stuck with the plain white screen. Ctrl+Alt+Backspace would make the desktop appear for an instant before dropping me back to the command line. The Resolution: Googling led me in a few different directions, and I tried things like trying to reset my xorg.conf via dpkg-reconfigure xserver-xorg. This generated a basic xorg.conf, but it also failed with this error about a battery: FATAL: Error inserting battery (/lib/modules/2.6.24-19-generic/kernel/drivers/acpi/battery.ko): No such device ??? This is a desktop! Obviously there's no battery! I never figured out what was up with that. Anyway, the new, very small xorg.conf didn't make anything different happen. I also tried updating a couple times along the way with: apt-get update apt-get upgrade Still no dice. Previous experience had taught me that, regardless of which OS you're using, if you have weird problems like this, there's a better-than-even chance that it's got something to do with the video drivers, so I thought I'd try updating those. The Unofficial Wiki for the ATI Linux Driver was very handy in this regard. I found it linked on the ATI driver page when I went there to download the driver. Of course, downloading and installing the driver yourself is considered the Hard Way, but I've done it once before, and it was the automatic way that got me in this predicament. The wiki has instructions for multiple version of multiple distros, including a page for Ubuntu 8.04. I followed the instructions along to the "Install .debs" step before I hit my first problem. The problem was that I had conflicting packages installed because while I was waiting for the driver to download, I had installed Envy with the intent of trying to have it straighten out my drivers for me. I never tried it though, opting instead to handle the installation myself. It turns out, though, that in installing it, several packages had been added that got in the way of the packages that the driver wanted to install. Removing the Envy packages let the driver packages install fine, but at the very end, I got this error message: [fglrx:firegl_init_module] *ERROR* firegl_stub_register failed Further Googling led me to a forum thread that mentioned that trying to install the fglrx module (the ATI drivers) with modprobe fglrx is what causes this error. I tried running that command, and sure enough, I got the same message. The thread also mentioned that modules "radeon" and "drm" seem to get in the way of fglrx, and removing them allows modprobe fglrx to work. lsmod showed that I also had the two modules in question installed, so I crossed my fingers (I had no idea what would happen) and removed them with rmmod radeon and rmmod drm, then ran the entire installation process again, just to be safe. This time it went fine. I started on my way down the checklist again, and when I got to sudo aticonfig --initial -f, I realized that this is something I should have tried in the first place, since this generates an xorg.conf file with all the right ATI stuff in it. Oh well. After finishing the checklist, my desktop is back, and this is the first blog post I've made from Ubuntu. Woohoo! P.S. The title of this post is a reference to one of the greatest movies ever: Dr. Strangelove.