Shotguns and Penguins

What is JSR 348 (JCP.next)?

2011-11-28T20:32:00.000-08:00

I was thinking about how to keep track of the articles I read and things I take time to learn about, like JSR 348, and I realized I should just stick it on my blog. That's kinda why it's here. So here we go: JSR 348 is nicknamed JCP.next because it revises the Java Community Process, which dictates how revisions to the Java platform are made. The final release was 18 October 2011, which means that every new JSR after that date is required to conform to the new requirements of JCP 2.8 (the new revision).

Generally, JSR 348 is focused on making sure in-progress JSRs are more transparent, are easier for anybody to get involved with, and keep moving at a reasonable pace so they don't stall out unfinished, like many have in the past. Just some of the measures include requiring that communication take place in a public forum, widening the pool of people who can be involved in JSRs to include just about anybody, and providing for removing uncooperative or unproductive (or just plain missing) members from the Expert Group of a JSR as well as replacing the Spec Lead if necessary. The Executive Committee was itself the Expert Group for JSR 348, and while working on the JSR, they followed all the guidelines that they themselves were putting into the new version of the JSR. Comments from the EC seem to indicate they felt it was a very positive experience.

Overall, it seems like an attempt initiated by Oracle to overcome some of the criticism after their takeover of Java from Sun and the fears that Java would become more closed and tightly owned by Oracle and possibly stagnate. From what I've seen while browsing around, there's a lot of hope that JSR 348 will be a success on those fronts. It only took a handful of months to complete and puts some pretty aggressive measures in place that look like they'll keep things open and moving along at a good pace. More importantly, it's gaining quite a lot of acceptance in the community, and it seems like people are breathing a sigh of relief at seeing Java looking like it's ready to come out of the quagmire it seemed to be in. Maybe the major releases will come along more often now and we can look forward to regular infusions of new language improvements?

A couple of the articles I read that are fairly short but give a good overview of the JSR:
The JCP Program Moves Towards a New Version: JCP.next
JCP 2.8 Ushers in a New Era of Complete Transparency

Effective Testing of a Jersey Resource Class

2011-04-20T19:56:00.000-07:00

Table of Contents

Iteration One: The Jersey Test Framework
Iteration Two: Customizing the JerseyTest
Iteration Three: Sneaking in a Mock
Iteration Four: Generate Mock Containers Dynamically

For a while now, I've been using Jersey for ReSTful web services with the resources classes and dependencies managed by Spring. I'm pretty avid about TDD, and a tricky question has been how to effectively test the JAX-RS resource classes. This post will take you on a brief journey through my evolving answer to this question. We'll examine the problems and considerations involved in testing as I describe several iterations of my testing approach, ending with my current approach, which I feel is very effective. Depending on your needs, you may find one of the earlier versions to be quite satisfactory.

Iteration One: The Jersey Test Framework

Just starting off, you're faced with the question of whether to just do pure unit tests of the resource classes or whether you should find a way to deploy them to a Jersey container so you can test the ReSTfulness of your resources. If you're choosing the former, then you probably wouldn't even be reading this post. If, on the other hand, you'd like to deploy your JAX-RS resource classes into Jersey and test the ReST calls thus created, then a quick search including the terms "test" and "Jersey" should point you toward the "Jersey Test Framework". I'm not going to cover that in this post, as it's fairly straightforward to set up and I know there are some other blogs out there that talk about it. Suffice it to say that the JerseyTest class from the framework lets you very easily start up a Grizzly container running your Jersey application that is very accessible to tests. Since I use Spring with Jersey, all I really have to do is pass in my existing Spring context files to load, and my Jersey app is running and ready to test. This is iteration one: your Jersey app deployed for testing just as it might be in production.

The first thing you'll notice when following this approach is that your test suite will quickly drag to a crawl. Especially as your app grows in complexity and number of tests, it will start taking longer to start up, and starting it up fresh for every single test method (the behavior of JerseyTest) is just out of the question. This led me to question of how often it's acceptable to start (or restart) the test container. It was the doorway leading to iteration two.

Iteration Two: Customizing the JerseyTest

On the matter of where and when to start a test container, once per test is far too often. Once per test suite is a little inflexible and would require some extra test code outside of the typical test class, which is all that I ever write. So I eventually decided that starting it once per test class wasn't too bad. You probably won't/shouldn't have a huge number of JAX-RS resource classes in one module/project anyway, and it's not likely you'd write more than one test class per resource; therefore, the number of startups needed to test a module will be kept fairly low. Additionally, it gives you the flexibility to configure the Jersey container specifically for each test class. Getting to this point just requires a little bit of extra test support code. That's the meat of iteration two: writing your own test support class to take control of the starting of the Jersey container. This class will ensure that the app is only started once and maintain the WebResource that's created by the JerseyTest to hand out to tests as needed. This is a fairly simple exercise that's left up to the reader, as there are far more interesting things to come.

As your project's complexity continues to grow, you'll begin to wonder about the wisdom of deploying your full application stack to the Jersey test container, and rightly so. First, it continues to slow down your tests by adding to startup time. More importantly, it can no longer be called anything like a unit test because in order to test your ReST calls, you'll need intimate knowledge of all that goes on under them, such as what constitutes valid input, possibly database constraints, and other preconditions that have to be satisfied in order for your ReST call to succeed. In short, your tests will become bloated, unreadable, and meaningless. Third, and possibly most insidious, if you're following your code metrics, a test of this type will give you all kinds of bogus test coverage. While you're only intending to test the JAX-RS resource class, it's going to be touching lines of code all through your codebase and contributing to code coverage without really asserting anything about those lines. It can give a real false sense of security. This brings us to iteration three, and the really interesting part of this post.

Iteration Three: Sneaking in a Mock

What can you do to test your JAX-RS resource deployed into Jersey without testing all the stuff that's normally under it in a real deployment? That sounds like a job for mocks! But wait! Our application is deployed inside a servlet container that's running in our tests, and there's no way to get access to stuff "inside" it. Specifically, since I just pass in a list of Spring XML files and Jersey starts the ApplicationContext itself, how can I inject a mock into the context? Its lifecycle is completely out of my hands. It's hidden away inside Jersey in the Grizzly container. Believe me. I dug around the source code for a few hours, and there's no easy way to get the Spring ApplicationContext out of there. So it's time to get a little inventive. First, let's make up some brief sample code. Suppose you have a resource class like this:

    @Path("/foo")
    class FooResource {
        @Autowired
        private FooService service;

        @GET
        @Path("{id}")
        @Produces(MediaType.APPLICATION_JSON)
        public Foo getFoo(@PathParam("id") int id) {
            return service.getFoo(id);
        }
    }

It depends on a service with this interface:

    interface FooService {
        Foo getFoo(int id);
    }

The question is how to get a mock of FooService that's available to your test wired into the FooResource bean, which is out of your test's control. There are certainly a number of possibilities out there, but I strive for simplicity. Here is the (first) simplest thing I could come up with:

    class FooTestService implements FooService {
        private static FooService delegate;

        public static void setDelegate(FooService delegate) {
            FooTestService.delegate = delegate;
        }

        @Override
        public Foo getFoo(int id) {
            if (delegate != null) {
                return delegate.getFoo(id);
            }
            return null;
        }
    }

This is a class that lives with your tests. It implements, and therefore is-a, FooService. (Note that TestFooService, while more readable, is a BAD name for it, as the Maven surefire plugin assumes that any class beginning or ending with "Test" is a test class and will fail the build because of your "test class" without any test methods in it!) Since it's a FooService, it's eligible for wiring into a FooResource. It has a static field and setter, meaning that no matter where an instance of this thing is created, as long as we're in the same JVM (ClassLoader, really), we can inject a value as the "delegate", making it available to all instances of the class, wherever they may be. The delegate receives all calls made to any instance of this class. The if statement is there to provide some flexibility in use: you don't have to provide a delegate if you don't want/need to. By creating a test-specific Spring context file for our test, we gain the ability to inject a mock into a Spring context that is otherwise out of our control:

    <!-- file FooResourceTest-context.xml -->
    <beans>
        <context:annotation-config/>
        <bean class="com.foo.FooResource"/>
        <bean class="com.foo.FooTestService"/>
    </beans>

With this, our test can now do something like:

    class FooResourceTest {
        private FooService fooService;

        @Before
        public void setUp() {
            // start the Grizzly container with our test-specific context file
            ensureGrizzlyIsRunning("/FooResourceTest-context.xml");
            // create a mock of the FooService
            fooService = mock(FooService.class);
            // inject the mock
            FooTestService.setDelegate(fooService);
        }

        @Test
            ...
        }
    }

...and voila! We're now testing the FooResource in isolation. We can test indirect outputs, such as verifying that it calls the appropriate service method(s) at the appropriate time(s). We can provide indirect inputs by making our service mock return different Foo objects or throw exceptions. We've taken care of the three issues I mentioned in iteration two: your tests have minimal startup costs, they're very narrow in scope, and they don't create a bunch of test coverage out of thin air.

Everything will be fine with this approach for a while, but then you'll find that it's so useful that these *TestService classes are springing up everywhere. You might even have multiple copies of them to serve different modules. I had three separate copies of a particular one in my project. For the benefit these things give you, the cost is really quite small. It takes a bit of effort to write the first time, but there's hardly anything to them, and they virtually maintain themselves since adding an interface method immediately requires you to add the method to your *TestService classes. Still, when you get to where you have about 20 of them, many of which are duplicates, you'll become convinced that there is a definite pattern here that needs to be pulled out of the crowd. It took a couple of weeks sitting in the back of my mind to bring all the pieces together, but I finally arrived at iteration four, which so far looks like the end of the line for this particular puzzle.

Iteration Four: Generate Mock Containers Dynamically

For the final, long-term solution to this problem, I was looking for a single, simple class that could provide the functionality of all of these *TestServices that were floating around. I knew it would involve some kind of dynamic proxying or similar magic. I thought I might use something from Spring to help out. Finally, in a moment of clarity, it all came together, and it looked a bit like this (the code isn't in front of me right now, and I'm not checking this with a compiler, much less running it):

    class DelegatingProxyFactoryBean implements FactoryBean {
        private static Map<Class, Object> proxiesByType = new HashMap<Class, Object>();
        private Class proxyInterface;

        public void setProxyInterface(Class proxyInterface) {
            this.proxyInterface = proxyInterface;
        }

        public static void addProxy(Class proxyInterface, Object delegate) {
            Object proxy = Proxy.newProxyInstance(
            DelegatingProxyFactoryBean.class.getClassLoader(),
            new Class[] { proxyInterface },
            new DelegatingInvocationHandler(delegate));
            proxiesByType.put(proxyInterface, proxy);
        }

        @Override
        public Object getObject() {
            return proxiesByType.get(proxyInterface);
        }

        @Override
        public Class<?> getObjectType() {
            return proxyInterface;
        }

        @Override
        public boolean isSingleton() {
            return true;
        }

        static class DelegatingInvocationHandler implements InvocationHandler {
            private Object delegate;

            public DelegatingInvocationHandler(Object delegate) {
                this.delegate = delegate;
            }

            @Override
            public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                try {
                    return method.invoke(delegate, args);
                } catch (InvocationTargetException e) {
                    throw e.getCause();
                }
            }
        }
    }

It's a fairly simple class, but it has some voodoo in it. Overall, it's just a FactoryBean. The layer of indirection provided by the factory pattern is just what we need so that we can configure one bean in a Spring context to be any type. The class combines static and non-static code in the same way the former FooTestService did to allow us to smuggle mock objects into a Spring context. Ultimately, the mocks become the output of the getObject() method, so they become beans in a Spring context that can be wired into other beans, just like we did in iteration 3.

The only code here that could be considered notable is in the addProxy() and invoke() methods. The addProxy method is analogous to the setDelegate() method of iteration 3. It's how we tell this thing about a mock that we want it to use and the interface for that mock--FooService in our previous example. When we give it the interface and mock, it creates a dynamic proxy based on the interface and stores the proxy away to be returned by the getObject() method later. The DelegatingInvocationHandler is what delegates method calls on the proxy to the mock. In iteration 3, we implemented all the interface methods ourselves, filling in the delegation code manually. Here, the invocation handler receives a method invocation and then turns around to invoke the same method on the mock object it was constructed with: no more code to write or maintain! The invoke method in the handler looks a little fishy because it catches and unwraps an exception. You might get a violation out of this if you run static analysis tools on it (throwing away a stacktrace). The reason is that when you invoke a method using reflection, if the invoked method throws an exception, it's wrapped in an InvocationTargetException. Since we want our proxies to behave exactly like the objects they're delegating to, we'll need to strip the outer InvocationTargetException from any exception being thrown. Here's what this thing looks like in use:

    <!-- file FooResourceTest-context.xml -->
    <beans>
        <context:annotation-config/>
        <bean class="com.foo.FooResource"/>
        <bean class="com.foo.DelegatingProxyFactoryBean">
            <property name="proxyInterface" value="com.foo.FooService"/>
        </bean>
    </beans>

    class FooResourceTest {
        private FooService fooService;

        @Before
        public void setUp() {
            // create a mock of the FooService
            fooService = mock(FooService.class);
            // inject the mock
            DelegatingProxyFactoryBean.addProxy(FooService.class, fooService);
            // start the Grizzly container with our test-specific context file
            ensureGrizzlyIsRunning("/FooResourceTest-context.xml");
        }

        @Test
            ...
        }
    }

The changes here from iteration three are trivial: in the context file, we add our FactoryBean and tell it to spit out a FooService. In the test, we do exactly the same steps as before, but in a different order. The FactoryBean as written requires that the proxies be added before the Spring context starts.

Now, you might have noticed that the dynamic proxies aren't strictly necessary here. They're just a reflection of what we wrote manually in iteration 3. You could remove that bit and just have the factory bean return the mocks directly; however, with a bit more effort, you can easily make the mocks optional and, at the same time, remove the requirement of setting the mocks before starting Spring. As written above, getObject() will return null if no proxy has been added for the type set on the FactoryBean instance, and that's not very friendly. Just add some code there to create a proxy that has no delegate, and modify the invocation handler to return some default values if there's no delegate to invoke. Now it can generate test proxies with some basic behavior without needing a mock. Then alter addProxy() to inject mocks into existing invocation handlers instead of always creating new ones. Problems solved.

That's the end of the road. This DelegatingProxyFactoryBean--with the enhancements just discussed--allowed me to remove around 20 test classes of the iteration three variety. Furthermore, you'll notice that there are no references at all to "test" or "mock" in that class. It's generic enough that it could perhaps find its way into other use cases.

One final caveat: as I mentioned at least a couple of times, I haven't tested any of this code. I wrote it all in notepad++ with a couple of references to some javadocs. If you find problems, let me know, and I'll update this post.

Tomatoes and Routers

2010-11-14T19:08:00.000-08:00

No, this isn't a candidate name for a new blog.
Over the past several months, during which my blog has been deafeningly silent, I've been writing RESTful web services based on Jersey and Spring 3 organized as a Maven project of around 20 modules. The entire project has an almost fully-automated build and release cycle with Hudson and Sonar running nightly code analyses. Our test coverage is about 87%, and we're aggressively pushing it higher. Cyclomatic complexity per method is 1.6. There are a ton of things I could blog about, but I just haven't taken the time to think through the distinct topics I should write about. This post is not about that.

This post is about my new Linksys WRT54GL. While I may be the last person on earth to get one of these (>3k customer reviews on Newegg? seriously??), I'm still going to give a brief synopsis, because it's just really cool. Yes, it's an old model--a dinosar in computer years. Yes, it only supports 802.11g. But respectively: there's a reason it's both the most- and highest-rated of all wireless routers on Newegg, and do you *really* need 802.11n? If you need the most speed possible out of your wireless network, then give this a pass and go with a wireless n product. If, like me, you never find yourself wishing for faster wireless--and you're at least 23% geek--not to say I'm limited to 23%--and not to say I'm not (how many points do I get for reading a Lisp book in my spare time?)--then this could be the perfect router for you.
Initial setup was a snap. The first thing out of the box was a paper with size 72 font that said "Put the CD in first". That was pretty much all the instructions, and the CD takes you step by step through everything in simple, well-presented steps accompanied by pretty pictures. With just that, you've got a number of powerful options for setting up your network, though maybe nothing you wouldn't find in competing products. What really gives this router its strength is the fact that it's a deliberately open platform:

"The Linux-based Wireless-G Linux Broadband Router was created specially for hobbyists and wireless aficionados." --http://homesupport.cisco.com/en-us/wireless/lbc/WRT54GL

There are numerous third-party firmwares available for it. From a fairly small amount of research, it seems that you can turn this router into virtually anything, as long as it fits in the available memory:

FTP server via ProFTPD
Windows file sharing client and/or server via Samba
Welcome/login page and access restrictions for network access for use in a public setting via Chillispot

And that's only a small sample of the stuff you can run on it using just the DD-WRT firmware.
That being said, I passed on DD-WRT and went with Tomato. It doesn't have all the capabilities, but the installation instructions for Tomato are very short and sweet. DD-WRT seems a bit more involved. Even so, with Tomato, you get static DHCP reservations (strangely missing from the stock firmware), internal DNS and DNS caching for your network (DHCP and DNS provided by Dnsmasq), SSH/telnet access to the router's OS--Linux, btw--Samba client for making storage available to the router, great visibility into your network usage via live graphs and historical metrics reporting, and a slick AJAX web UI.
All of that, and installing Tomato really was as simple as downloading it and using the router's standard "update firmware" screen to install it! If you don't already have a WRT54GL, buy one. Or maybe two. If you do have one but haven't switched to a third-party firmware, do it now! You won't regret it! Unless your power goes out in the middle of flashing!

Best Salsa Ever

2010-08-14T20:04:00.000-07:00

My family got this salsa recipe from a little, then-hole-in-the-wall Mexican food restaurant in San Antonio, TX somewhere around 25 years ago. It's still the best salsa I've found anywhere, and the restaurant--now successful and with multiple locations--is still making the same salsa. Naturally, our version and the restaurant's have diverged over the years, but the ground rules have never really changed.

First, a disclaimer: I don't actually have or follow a recipe! I just know the ingredients that are needed, and I put them together until it tastes right. What I'm about to give you are the guidelines I use for getting all the ingredients and some rough guesses at the right quantities to make it all work together. If you follow this recipe exactly, you'll probably (I think) have something edible. If you want it to really shine, you'll need to take your time and pay attention to the "tuning" notes I've added at the end of this post. You'll also probably need to make it several times before you get the hang of it. What makes it difficult is that the amounts of ingredients you need change based on variations in the quality and flavor of the ingredients (primarily the tomatoes) that you get. Now, on with the show...

The Ingredients

I've split the list into "main" ingredients, which are what make up the body of the salsa, and "flavor" ingredients, which you use to tune the flavor to your personal preference. The quantities--or at least the ratios--of the main ingredients that I use are pretty well-known (by me, that is):

Main Ingredients

8 lb Roma tomatoes
2 lb slicing tomatoes
1 lb Serrano peppers
1 lb sweet onions

Note: With these quantities of main ingredients, you'll end up with very near the right amount of salsa to serve about 50 people at a rehearsal dinner where taco salad is being served (ask me how I know!) It probably makes around six or eight quarts. Make as much or as little as you want. The important part is to get the ratios right. You need about 10:1 by weight tomatoes to peppers and roughly the same amount of onion as pepper. In other words, one pound of peppers for ten pounds of tomatoes or 1/4 pound of peppers for 2.5 pounds of tomatoes. The onion should always be roughly the same as the peppers. If you cut back on the main ingredients, remember to scale back on the flavor as well!

Here's where it gets tricky. I've tried to guess at rough quantities here based on the quantities of main ingredients I listed above, but this is just that: a guess. Take it as such! I'll cover these ingredients more in depth in the step-by-step directions later:

Flavor Ingredients

1/3 cu lemon juice (bottled recommended)
1/3 cu lime juice (bottled recommended)
3 T garlic powder
3 T salt

The Tools

Gather up the following tools and take them to the place where you'll be working:

Cutting board
Knife--sharp and/or serrated
Tomato corer--great for coring the tomatoes but even better for deseeding peppers
Small bowl or nearby trash can for scraps
Two large bowls, each big enough to at least hold all the tomatoes--you can get by with only one, but two is better
Food processor--NOT a blender!
Spatula for scraping/stirring
Dish towel--spill/splash cleanup
One or two powder-free latex gloves--for protecting your hand(s) from the peppers
Chips for sampling the salsa to fine-tune the flavor

The Directions

Pop the stems off the tops of the peppers and wash the tomatoes and peppers.
Protecting your hands with the gloves, halve and deseed the peppers. Taking the seeds out greatly reduces the heat from the peppers. The tomato corer is great for removing seeds from pepper halves. If you're feeling brave, skip this step. If you don't use gloves, you'll want to refrain from touching your eyes, nose, or any other sensitive areas for the next day or two.
Core the tomatoes, and take off any other blemishes, bad spots, etc. Quarter the romas. As the slicing tomatoes are larger, you'll probably want to cut them into six or eight pieces. Put the tomato slices into one of the large bowls. If the tomatoes are excessively juicy, this lets the juice drain from them while they're waiting to be chopped up.
On to the processing: chop up the peppers in the food processor pretty finely. You don't want it pureed, like a blender, but it's hard to get the peppers too fine in a food processor. High speed is fine for peppers.
Cut the ends off of and peel the outer skin from the onions. Slice them up and chop them in the food processor, too. I save chopping the onions until right before I put them in to spare myself and others from the fumes. If your food processor is big enough, you can do the onions and peppers at the same time. I try to chop the onions less finely than the peppers, but that's not too important, either. High speed is fine here, too.
Put the chopped peppers and onions in the second large bowl and cover them unless you want to drive everyone from the room you're working in.
Now run the tomatoes through the food processor. You want to leave the tomatoes in larger chunks than the peppers: around 1/4 inch in size or larger. This may take some practice in your food processor. If you have a sharp blade, it's very easy to reduce the tomatoes to mush. Even the low speed of dual-speed food processors may be too fast. If so, you'll need to use the "momentary" setting and quickly flip it on and off to slowly dice up the tomatoes. As soon as you have no chunks left that are larger than about 1/2 inch, they're done. After chopping the tomatoes, pour them on top of the onions and peppers in that bowl.
After all the tomatoes are chopped, you'll have some tomato juice in the bottom of the bowl that was holding them. You can add this if you like, but note that the salsa tends to float, so when you get to the bottom of the bowl, it'll be mostly juice. I usually just dump the extra tomato juice.
Now for the hard part: the flavor ingredients. If you want to follow this recipe exactly (not recommended!), just add the lemon, lime, salt, and garlic in the amounts I listed and stir well with the spatula. Keep refrigerated, and let me know how it turns out. If you want truly great salsa, go on to the following paragraphs to learn the basics of how to balance the flavor ingredients with the main ingredients.

Fine Tuning

Instead of the full amount, start off by adding about three quarters of the amount of the flavor ingredients I listed, stir well, and then taste it. The exact amount isn't important. Just start slow and build up. Use the following guidelines to decide what to add:

If it tastes "flat", or too much like tomato, add more salt.
If it doesn't taste sweet enough, add some garlic and citrus (lemon and/or lime--your preference). Sometimes a touch of cumin, aka comino, will bring out the sweetness, too. Only use a little--like less that 1/4 teaspoon per cup of salsa. Some restaurants go crazy on the cumin in their salsa, and it just overwhelms everything else.
If it seems too hot, add citrus, especially lemon. It reduces the peppers' bite. Don't go overboard, though!
If you get too much lemon in, it can be balanced with salt up to a point. (Not sure if the same works for lime.)
If you get too much salt, citrus may help a little bit, but you'll probably have to pick up some more tomatoes to reduce the saltiness. A little bit of sugar or cumin can help here, too, mostly by masking the salt flavor, but I don't recommend it unless you've only gone a little bit over on the salt.
If it just doesn't taste quite "right", it probably needs more garlic. I can't help you any more there. I guess that one just comes with experience.
If you get to a point where you think you're adding too much of the flavor ingredients, and it still doesn't taste quite right, letting it sit for 1-2 hours--I recommend refrigerating ASAP--can change its flavor quite a bit as the ingredients blend. Just check it again later.

Additional Notes

If, while stirring the salsa, some light yellow foam forms on top, it means you're pretty close to the right mix of ingredients. I'm not sure what does that, but it always happens.

When made from good-quality ingredients, this salsa will keep for as long as 3 weeks in the refrigerator, but it's best if used within the first week or so. Freezing this stuff is a no-go. It basically turns to water with little scraps of tomato floating in it. Canning preserves it better than freezing, but don't attempt to can it unless you're familiar with the issues around safe home canning of tomatoes!! I've only canned it for one growing season so far, and the result was quite different from the fresh product, but it turned out well enough. I intend to keep working on it to see what I can do with it.

Some people like to add cilantro to their salsa. I admit that it does add a pleasant touch, but I generally don't find it to be worth the hassle. Cumin is another popular addition, but be careful: a little goes a long way.

You can use fresh lemons, limes, and garlic. If you know a fresher source of salt than the little cardboard container, let me know. Using all fresh ingredients can give the salsa a fuller flavor, but the flavor of citrus fruit and the potency of garlic varies widely from one store visit to the next. This makes it even more difficult to get the right blend of ingredients, and lemons with the wrong flavor can make a whole batch turn out poorly. That's why I nearly always use bottled lemon and lime juice and garlic powder. These products have a fairly uniform flavor and potency, so it's easier to get consistently good results with them.

Many salsa recipes call for Jalapeño peppers instead of Serannos, and if you enjoy the taste of Jalapeños you can use some here. Try using 50% Serrano and 50% Jalapeño. Just keep the tomato:pepper ratio correct, and don't leave out the Serranos entirely. They're part of the key to this recipe.

You may also be tempted to try different types of onion. I've tried a few, and I've never found any to be satisfactory. In fact, if I can't get sweet onions, I sometimes just go without any onion at all. Ideally, find some Texas 1015's.

On the other hand, feel free to experiment with different varieties of tomatoes. I list Roma and slicing tomatoes because that's generally what's easy to find at the grocery store, and using Romas as filler is cheaper than going with all slicing tomatoes; however, I've experimented with several other varieties of tomatoes including Phoenix, Celebrity, Better Boy, Beefsteak, and BHN 444. All of them made at least a decent salsa except for the Beefsteak. I didn't like the taste of that at all. Better Boy and Phoenix are my personal favorites so far. YMMV. Note that different tomato varieties will need different combinations of flavor ingredients--another complication!

Setting Up a Thin Client Network with LTSP

2010-03-07T18:47:00.000-08:00

I've become interested in thin client software as a potential way to use a cheap laptop as a mobile GUI for applications running on a more powerful desktop or server. Yesterday, I tried out one called the Linux Terminal Server Project (LTSP). I found out it's distributed with Edubuntu as a way for colleges and universities to set up thin client labs. I just wanted to see if I could get it working, and I was pleasantly surprised by how easy it was. My steps:

Install LTSP server

With VirtualBox, I created a VM with 512M RAM and a 5G dynamically sized hard drive. Following the Ubuntu Community Documentation, I downloaded the Ubuntu 9.10 alternate amd64 ISO to install the LTSP server on it. It was pretty simple: choose "Install an LTSP server" from the Modes menu, then "Install Ubuntu".

Realize I was supposed to have two network adapters

One thing I could've done differently to make things simpler later on: add a second network adapter to the VM. LTSP server wants two NICs: one for communicating with the outside world and one for communicating with the network where the clients are. There's supposed to be a way to configure it with just one NIC, but I belatedly added the second one and configured it myself rather than puzzling that out. The first adapter had the VirtualBox default setting of "NAT", which lets it communicate out through the host interface. When I created the second one, I set it to "Internal Network" so that it could only communicate with other VMs, where my client would be.

Make LTSP DHCP work with new adapter

LTSP server runs a DHCP server on the client-facing network interface. This is the first part of how LTSP clients start up: they get an IP address and some other instructions from the DHCP server on the LTSP server. Because I had only the one NIC at first, the DHCP server wouldn't start, either, and I had to figure out right network settings to make DHCP run on the second, "Internal" adapter.

Create LTSP client VM

After the DHCP server was working, I created a second VM with 512M RAM and no hard drive. I set its sole network adapter to "Internal Network" so that it would be able to see the second adapter of the LTSP server, where the DHCP server was running. Then I set it to boot only from the network and started it up. It almost immediately got a response from the DHCP server, but it failed with this error:

PXE-T01: File not found
PXE-E3B: TFTP Error - File not found.

Fix bad path in LTSP DHCP configuration

This one took me some research to figure out, but it turns out that it's just a simple path problem. In the DHCP configuration of the LTSP server, there's a line that gives a path to a file called "pxeconfig.0", which is the initial file that has to be retrieved for a network boot to work. The server tells the client via DHCP where that file lives, and the client uses TFTP, "trivial file transfer protocol", to retrieve it. For some reason, the path to the file was wrong when I installed it. I don't remember what it was set to at first, and I don't have the correct value in front of me. It was something like /var/lib/tftpboot/ltsp/amd64/pxelinux.0. Also, something does some "chroot"ing somewhere, making the correct path for requesting the file /ltsp/amd64/pxelinux.0 (I think). As soon as I got that right and restarted the DHCP server, I was good to go.

Create a user with a strong enough password to suit LTSP

One last thing: after fixing the path to pxelinux.0, the diskless VM was able to get all its required files and boot to a login screen, but I couldn't log in to the client with the same credentials I was using on the server. I had created a user with a trivial password because this was just a test VM, and I read something somewhere about LTSP having different password requirements than Ubuntu, so I made another user with a stronger password, and then I could log in. That's it. Now I can start the client VM, which boots from the network, and get to a Gnome desktop. Everything I run displays on the client but runs and stores files on the server. Pretty neat.

Managing Unmapped Tables with Hibernate

2010-02-03T19:33:00.000-08:00

I've published a new post over on codehangover.com explaining how you can use Hibernate's "auxiliary database objects" to simplify managing your database schema during development and testing. Check it out.

Git Quick Reference

2010-01-22T18:25:00.000-08:00

I've been lazy/sick/on vacation for a while, but I think I'm finally ready to release my Git Quick Reference into the wild. It's kind of a follow-up to my series of Git tutorial posts, collecting all the important stuff into one, relatively short document for easy access once you've started down the road of learning your way around Git. It's available on Scribd: Git Quick Reference

The Journey to Git, Part X—Communicating Between Repositories

2009-09-06T20:31:00.000-07:00

So you want to do some collaboration using Git. If you don't know where to start, you're in the right place. Start here. This post, like my earlier Git posts, will take you on a guided tour of how to collaborate with others (or yourself) using Git remoting. It will be light on theory and practical application of principles and instead focus on the "how" so you can start using it as quickly as possible.

In this post, I assume you're comfortable working with a single Git repository with the basic commands like "git add", "git commit", "git branch", "git merge", and so on. If you're not to that point yet, hop back to my earlier posts in this series for a quick walkthrough:

Making a Clone

We need an existing repository to start from, so create a directory named "cloneme", change to it, and set up a repository like so:

git init
echo "foo" > foo
git add foo
git commit -m "first commit"

Simple enough: a repository with one commit and one file being tracked. Now move to the parent directory of cloneme, and run:

git clone file:///path-to-cloneme clone

Note: The "/path-to-cloneme" part should be the absolute path to the cloneme directory. It's best to go absolute here for a couple of reasons. Don't use a relative path unless you understand the implications of having a relative path stored in your .git/config file.

You've just performed your first remote Git operation by cloning an existing repository. As you might expect, you now have a complete copy of the "cloneme" project in the "clone" project. Note, however, that it's not just a copy of the working tree. It's a complete clone of the original repository. Git is, after all, a distributed VCS.

All we did in this first clone was basically a filesystem copy since we used the "file://" transport. Git, of course, supports remote operations over networks with other transports: ssh, rsync, http, https, and a native "git" transport. Each has its own, very similar, URL syntax for specifying how to find a remote repository. I use the ssh transport almost exclusively. It's secure and just as easy to use as the file transport.

At this point, you have two repositories with identical content. Running "git log" in both of them, for instance, would produce identical output. Start up gitk now, and you'll see the familiar "master" designator pointing at the head of the branch, but next to it is another thing that says "remotes/origin/master". The initial "remotes" is kind of a namespace that's set aside for specifying branches that are in remote repositories. The next piece, "origin", is the name of the remote repository, and the final one is the name of the branch in that remote repository. When you clone a repository, the cloned one automatically becomes the "origin" for the clone, making for convenient interaction with it, as we'll see in a moment.

What this gitk output is telling you is that the head of the remote repository's master branch is at the same commit as your local master branch... as far as this repo knows. Changes in a remote repository are not automatically detected by gitk, so something in the remote could've changed, but gitk won't reflect it until you "git fetch" it. Let's take a look.

Getting New Changes from the Origin Repo

Go back to cloneme, and make a new commit:

echo bar >> foo
git commit -am "second commit"

Now go back to clone. Both "git log" and gitk will show exactly the same thing as before. As I mentioned, these two commands don't do any remoting, so they have no way of knowing about the change. In order to see the new commit, you need to fetch it:

git fetch

When run with no arguments, this command will retrieve all of the latest changes from the remote repository named "origin". That's some of the convenience that I mentioned earlier. Run gitk again, but this time with "gitk --all", or you'll only see a partial picture. Now you can clearly see that the remote named "origin", which is cloneme, is one commit ahead of clone.

Note: When I say "all of the latest changes", I do mean "all". In this exercise, we're confining our work to a single branch, but "git fetch" retrieves the latest changes from all of the branches of the specified remote, as well as any new branches that have been created.

Next run:

git status

You'll see that it also quite clearly tells you that "origin" is ahead of you with a message like:

Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.

Let's go ahead and do the mentioned "fast forward":

git merge remotes/origin/master

That should seem pretty natural to you. It's just a simple fast-forward merge, the same as you'd use to merge any branch into another. The only difference is that you're effectively merging changes from a remote branch into a local one.

Local and Remote Branches

This is a good place to take a look at exactly what that "remotes/origin/master" thing is. Run:

git branch -a

You should see output like:

* master
  origin/HEAD
  origin/master

The -a flag to "git branch" tells the command to display both local and remote-tracking branches, which is what remotes/origin/master--shown as "origin/master" here--is. It's a local representation of a remote branch. A remote-tracking branch exists for the sole purpose of storing commits that you fetch from remote repositories. You don't ever make any commits or do anything else to them except for fetch remote changes into them.

You can, however, make a local branch that "tracks" a remote-tracking branch and make commits there. We'll get into the details of that later, but you already have one of these. The master branch of the repository in "clone" is a local branch that tracks the remotes/origin/master remote-tracking branch. It was set up this way when you did the clone. That's how Git was able to tell you that you were a commit behind the remote branch. It knows that your local branch "master" is tracking a branch named "master" in the remote named "origin".

The Fast Way: Pull

The fetch and merge are fine for illustrating what's happening, but generally you just want to pull the latest changes from the remote repository directly into your local branch, and the two separate commands are an unnecessary step. Enter "git pull". This command is nothing but a combination of "git fetch" and "git merge". It's even clever enough to figure out what you want it to do without any arguments if you're on a branch that's tracking a remote-tracking branch, like your master branch in "clone". Go make another commit in "cloneme":

echo baz >> foo
git commit -am "third commit"

Now switch to "clone" and simply run:

git pull

Everything happens automatically, and the "master" branch of "clone" now has the new commit in it. As I mentioned, you didn't have to tell "git pull" which branch to merge from because the current branch, "master", tracks "remotes/origin/master", so that's the one it selects for the merge.

Note: Unlike "git fetch", "git pull" doesn't pull all changes from all branches into the matching local branches. Since part of a pull is a fetch, it does fetch all of the changes into the remote-tracking branches, but only the current local branch is updated with changes from its respective remote-tracking branch. That is, only one merge is performed upon a pull.

Everything so far has just been in one direction: from the original repository to the clone. Eventually, you'll want to go back in the other direction. Make a fourth commit in the "clone" project:

echo clone >> foo
git commit -am "commit in clone"

Now switch to the "cloneme" project. When you clone a repository, the cloned one doesn't gain any knowledge of the clone, so it should be no surprise that running a simple "git pull" from "cloneme" will get you an error like:

fatal: 'origin': unable to chdir or not a git archive
fatal: The remote end hung up unexpectedly

Detour: Configuring a New Remote Repository

Remember that "git pull" tries to fetch changes from "origin" if you don't tell it something different. Because this repository wasn't cloned from anything, it doesn't have an "origin". We'll need to tell it where it can get changes from by adding a remote repository:

git remote add theclone file:///path-to-clone

Note: Again, /path-to-clone should be the absolute path to the "clone" project.

This adds a remote named "theclone" to this repository's configuration.

Pull Continued

With the newly configured remote, pulling changes is as simple as:

git pull theclone master

Why the extra arguments? Well, first, we have to specify the name of the remote, since the default is "origin". We could have named our remote "origin", but that's not really what it is, so I picked something else. As for the "master" part, since our current branch--the local branch "master"--isn't set up to track any remote-tracking branches, "git pull" doesn't have any information about which remote branch to merge changes from. Therefore, we explicitly state which branch we want to use. The changes are pulled into the current branch.

You now know how to clone repositories, add remotes, and pull changes. That's about all you need to know to start using Git to collaborate on projects; however, there's one more thing that Git lets you do: push. Because of what it does, it's somewhat more difficult to use correctly. There are some caveats, which I'll mention as we go along.

Pushing Changes Instead of Pulling

When would you need to push changes out instead of pulling them in? Well, it's great that Git is distributed and that everyone has their own complete repository for working in, but if you were working on a project team of even moderate size, you can imagine how difficult it would be to say what the "current" state of the project is if everybody just has their own repos and swaps changes at will. You would want to create what Git terms a "blessed" repository. That's a repository where finished work gets pushed to and where you pull from to get the latest "official" state of the project.

Warning--Angels Fear This

Before we go on, let me clearly state that the Git FAQ says you should only push to a bare repository "until you know what you are doing". A bare repository is one that was created with the --bare option. It has no working tree. It says this because pushing into a branch that is checked out to a working tree can be problematic. That's what we're going to do here, though, because properly managed, it's not an issue, and I find it to be very useful to sync changes between two different computers that I'm working on. Just realize that the issues we'll encounter related to working tree state don't arise when you follow the FAQ's advice of pushing only to bare repos.

The Simple Push

At this point, your two repositories, "cloneme" and "clone" should be in sync. That is, they both have the same set of four commits in them. A "git pull" from either side will end with an "Already up-to-date", and neither has any uncommitted changes. Let's add a new commit to "cloneme" and push it to "clone":

echo pushme >> foo
git commit -am "a commit to be pushed"
git push theclone

The first thing to note is that we didn't specify a branch name, only the name of the remote. When you do that, changes in all local branches are pushed to the remote if a branch with the same name already exists there. In other words, if we were to create a new branch named "mybranch" in project "cloneme" and run "git push theclone" again, no changes would be made because that branch doesn't exist in "clone". If you want to send the new branch across, you could do it by specifying the branch name like "git push theclone mybranch".

Why Push Isn't So Simple

Let's go see what "clone" looks like now. You might be a bit surprised at the result. A "git log" will show you that the latest commit was pushed successfully. However, "git status" shows that you have changes in your index. How did this happen? It was clean before the push. Well, run a "git diff --staged" to see what it says has changed. You should see something like this:

diff --git a/foo b/foo
index 5a347e2..90c3f45 100644
--- a/foo
+++ b/foo
@@ -2,4 +2,3 @@ foo
 bar
 baz
 clone
-pushme

It's saying that in project "clone", you've removed the line that you just added in "cloneme". Why? Because "git push" does not make any changes to the working tree or index of a remote repository, lest work be lost. Particularly when you push to a remote that's not in your control, you have no way of knowing whether somebody else is making changes to the working tree or index at the same time, and you can imagine the havoc if "git push" were to mess with those changes. So while the new commit was added to the repo, the working tree hasn't been touched, and is in the same state as it was when the HEAD^ commit was the latest. Therefore a "git diff" shows exactly that: the output you would expect from running "git diff HEAD HEAD^" in either of the repositories.

To correct this, since you know that no work will be lost, simply run:

git reset --hard

Now your working tree and index properly reflect the tip of the branch, where you want them to be.

Another Restriction on Push

There's one more caveat about "git push": by default, it will only succeed if you can fast-forward the remote branch(es) you're pushing to. Put another way, if you're pushing from "cloneme" master to "clone" master, then the set of commits in "cloneme" must be a superset of the ones in "clone", or the push can't succeed. Again, it's a question of overwriting someone else's work. The most likely way for this to happen is if you're trying to push changes to a remote branch that you previously pulled from, but someone else has added new commits to it in the meantime. The solution in that case is to do another "git pull" to get the latest changes, and then you'll be able to push because you'll have the required superset of commits.

Of course, you can force Git to do a non-fast-forward push. Just make sure you understand that this will destroy work that's been done! Let's look at an example. In project "clone", make a new commit:

echo loseme >> foo
git commit -am "this commit will be lost by a bad push"

Now go back to "cloneme" and run:

echo destroyer >> foo
git commit -am "this commit will cause the loss of a commit in clone"

First, try a typical push:

git push theclone

It will result in an error like:

 ! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to 'file:///cygdrive/c/dev/projects/clone'

Now force it to do the push with:

git push theclone +master

The '+' indicates that Git should force the push. Go over to "clone" now, and a "git log" will show you that the last commit we made there has disappeared. Because we're pushing to a non-bare repository, the index will still have the lost change in it, but another "git reset --hard" will bring it up to date with the repo.

Finally!

And that, as they say, is that. Journey complete. If you've read and followed along with all of my Git posts, you may be an incurable geek, and you certainly should know enough to be dangerous with Git and to start seeing how great it is in comparison with a centralized VCS. Aside from a quick command reference, which is almost finished, this is all I plan to post about Git for the time being (finally!!! woohoo!!!). If you have any questions, feel free to drop me a comment, and I'll answer it to the best of my ability.

Late addition: I've published a Git reference card on Scribd that should be good for reminding you of the commands you need to use without having to dig back through these posts.

Book Review: xUnit Test Patterns + Code Hangover

2009-08-21T21:49:00.000-07:00

This is not a book review.

This is a book review.

Over the past few weeks, I read another book: xUnit Test Patterns. I posted the review on a different blog: codehangover.com. It's a new blog that I'm coauthoring with some former coworkers of mine. I haven't decided exactly what the division of labor between this and that blog will be, but I intend to put the more formal ones, like book reviews and my Git series (I'll finish it soon!), over there.

Some of the other authors on codehangover.com had their own technical blogs, and we decided to combine efforts to hopefully make a more useful blog and, honestly, one that will draw more traffic and maybe earn us all a bit more from affiliate sales ;)

The Journey to Git, Part IX--Communicating from Git to Subversion

2009-08-07T19:15:00.000-07:00

In this second part of the Git/Subversion interaction guide, we'll explore the commands that let you do the equivalent of "svn update" and "svn commit". You need to already have a Git repository that's linked to a Subversion repository. The previous post in this series will help you with that if you need it.

Getting a Git copy of a Subversion repository and making local commits/branches/whatever in it is great. How do you get further updates of commits that others have made to Subversion? What do you do when you're ready to send your changes back to Subversion? First, decide which branch it is you want to send or receive commits for, and make sure it's checked out. Both the commands I'm going to discuss work within your current branch.

Before committing anything to Subversion, it's never a bad idea to update first and see if there have been any changes, so let's look at that command first.

Updating from Subversion with Git

The Rebase

It may not seem apparent at first, but when we pull changes from SVN to Git, what we really want is a rebase. Why? To maintain a perfectly linear history for the sake of Subversion. We've seen the git rebase command before. Recall that it's the one that lets you freely squash, edit, delete, and move commits around in a branch. To pull the latest changes from Subversion into your current branch--every Git branch can be traced back to a Subversion branch from which it originated, and that's the one it pulls commits from--you first need a clean index and working tree, and then run:

git svn rebase

Note: See the note at the beginning of my previous post to learn how "git svn rebase" is currently broken in Cygwin and to find a workaround.

Your index and working tree have to be clean because of what the rebase does, which I'll get to in a minute. First, let's examine why we have to use a rebase more in depth. When you're working out of a Subversion repository with Git, you'll always be building on top of Subversion commits. Say you're on a branch where there are two Subversion commits: A and B. Then you make commit C in Git locally. Meanwhile, someone else has made a new commit to Subversion--call it X. When you go to pull that change from Subversion down to your local Git repo, where should it go? Your first inclination might be that your local history should become A -> B -> C -> X, but this is 100% wrong. Subversion already has A -> B -> X, and you're not easily going to convince it to put C in front of X. Git, on the other hand, has no problem at all sticking the X before the C. That's exactly what "git rebase" is for. Therefore, when you pull commit X from Subversion, you want to end up with A -> B -> X -> C. That is, you want Subversion commits to all be on top of each other and ahead of any of your local commits.

Now, a "git svn rebase" behaves much like a normal "git rebase". First, it moves all of your local commits--the ones that Subversion doesn't know about yet--out of the way, effectively taking them out of the branch and making HEAD point at the latest SVN commit that's in your Git repo. Then it pulls down all of the SVN commits that aren't represented locally and applies them one at a time on the HEAD of the current branch. This part should all go smoothly because it's basically just copying history from Subversion into Git. After that's done, the rebase puts your commits back on to the HEAD of the current branch, again one at a time.

When Conflict Occurs

When your commits are being reapplied, it's quite possible that you'll experience conflicts with the new commits you got from Subversion, just as you would with "svn update", and this is the main reason that your working tree and index must be clean before you start any kind of rebase. In the event of a conflict, you use your working tree and index to resolve it.

It's very important that you pay attention to Git's messages during any kind of rebase because if there is a conflict, it becomes an interactive process. The rebase will stop, and you'll be looking at a dirty working tree with unmerged files and possibly some staged changes in the index. All the changes you see are the content of the commit Git was trying to apply when the conflict happened. It's waiting for you to resolve the conflict somehow and then tell it to continue the rebase with:

git rebase --continue

Conflict resolution works exactly as I described in my post on merging. Note that we continue the rebase with the "git rebase" command and not "git svn rebase". Once the rebase is kicked off, it acts just like any other rebase you'd perform in Git, and as with any other rebase, resolving the conflicts and continuing is only one of your three options. You can also skip the current commit with:

git rebase --skip

You'd use this if, for instance, the current commit is no longer applicable because of an upstream commit that you've received. The commit is effectively deleted, and it won't be in the branch when the rebase is complete. The third option is to abort the rebase entirely with:

git rebase --abort

You can always abort up until the rebase is completely finished. An abort takes you back to the state you had before you started the rebase. If you really get yourself in a bind, or if you decide you just don't have the knowledge to resolve the conflicts effectively, you can always abort and come back to it later.

The really important thing about rebasing is that when it stops in the middle, you must see it through to the end one way or another. Don't go off to work on something else until the rebase is complete, or you're really going to confuse yourself.

Now that we've got all the incoming changes, let's see how to send changes back to Subversion.

Commiting to Subversion with Git

Compared to what we've seen so far about Git-SVN interaction, sending your local commits back to SVN is a breeze. Just make sure you have a clean working tree and index, then run:

git svn dcommit

There's not much that can go wrong with this command. The working tree and index have to be clean because a dcommit ends with a rebase or a reset, and we've already seen why you have to clean those up before a rebase. I'm not certain exactly what the rebase/reset does, but I think it has to do with putting the "SVN version" of the commit in your branch in place of your local commit.

When the dcommit is complete, there will be a commit in Subversion matching each of the local commits you had in Git, and the commits in Git will all reflect a git-svn-id, indicating that they're recorded in Subversion.

That's it. Two commands are all you need to swap commits with a Subversion repository. Next up, I'll talk about interacting with remote Git repositories. You'll find that the basics are quite similar to the Subversion interaction, but it's a lot more powerful.

The Journey to Git, Part VIII--Connecting Git to Subversion

2009-08-07T18:28:00.000-07:00

In this post, we'll start seeing how to use Git as a client to a Subversion repository. This is an excellent way to get your feet wet with Git without forcing the learning curve on others working on the same project. It might also be a useful intermediate step in moving from SVN to Git by getting all the members of a team accustomed to Git while still having their old SVN client as backup in case they get lost. As has happened previously, when I got to the end of what I thought was one post, I decided it was way too long, so I'm breaking it up into two pieces. This piece discusses cloning an existing Subversion repo and what you'll have after you do that. The next one explains the commands you use to trade commits with Subversion: the equivalents of "svn commit" and "svn update".

Before you dig in here, you should be able to use basic Git commands like commit, checkout, and branch. If you're not comfortable with that, have a look at my earlier posts:

Setting Up for Subversion Interaction

All Subversion interaction is done through a set of special sub-commands that start with "git svn". If you're running Git through Cygwin, there's two things to note. (Non-Cygwin-users can skip to the next paragraph.) First, you need to install the "subversion-perl" package in Cygwin to be able to use the "git svn" set of commands. Second, a change introduced to Cygwin a few months ago slightly broke "git svn" under Cygwin. See this message for a description of the problem and a workaround for it. When I refer to the "git svn rebase" command in the next post, you'll need to use the mentioned workaround in its place. It may also affect the "git clone" command, but I've neither checked it for myself nor seen any reports on it.

Only Cygwin Git users need to do anything special. On Linux and in mysysgit, everything is already in place. I expect the primary way that Java developers will start using Git is by cloning an existing SVN repo, and that's what I'm going to go through here.

Cloning an Existing Subversion Repo

To use Git to work against an existing SVN repository, your first step is to clone it. Remember that Git is a Distributed VCS, meaning you have your own copy of the entire repository. Cloning SVN is one way to get one.

Note: This can take several hours on a large project with a moderate number of branches because of the way SVN stores branches and the sheer number of files that have to be transferred!

If you're ready to start the clone, get the URL of your SVN repo, and switch to the directory in which you want your project directory to live. The "git clone" command creates a subdirectory and checks out the project in it. For a Subversion repo using the standard directory layout--that is, directories named trunk, branches, and tags--run:

git svn clone --stdlayout --username=<your username> <svn url> foo

If your repository structure differs from the standard layout, use this form instead:

git svn clone --trunk=<trunk dir> --tags=<tags dir> --branches=<branches dir> --username=<your username> <svn url> foo

Username is only required if using authentication, obviously, and "foo" is the name of the directory to create to hold the project. If you don't provide this, then the last bit of the URL--after the final '/'--will be used as the directory name.

Working with a Git Clone of a Subversion Repository

After a successful clone, the target directory will have a typical Git repository and working tree in it. Your standard master branch will be there, and master's HEAD is what's checked out. In my experience, the content of master will be the Subversion branch with the most recent commit on it, but I haven't seen this behavior documented anywhere. You can see all of your Subversion branches and tags by running:

git branch -r

Note: I haven't covered remote Git interaction yet, so this may stray a bit into unfamiliar territory. Just understand that master is your local branch, where you do your work. If you were to "git commit" something, this is where that commit would go. All the Subversion branches you just saw are called "remote-tracking branches". For the most part, you can pretend they're not there. They just act kind of like a mirror of the Subversion repo so that you always have a copy of it around. The usefulness of this will become apparent later. Finally, master "tracks" one of the remote Subversion branches, meaning initially it contains exactly the same commits as that branch, and when you commit to or update from Subversion, that's the branch you'll be interacting with.

So now you have a Git copy of your SVN repo. What next? Well, now you can develop away using Git just like you always would: make commits, branch, merge, rebase, etc. There's just one caveat: don't fool with the history that came from Subversion. Don't try to rebase and change SVN commits around, for example. Just treat the commits from SVN as read-only. Immutable. Untouchable. Get the idea? If you screw with SVN's tiny brain in that way, don't come back to me unless it's only to describe how your SVN or Git repo melted down! I'd be interested to hear about that. You'll know the SVN commits because when you "git log", you'll see a special "git-svn-id" line in each SVN commit. Other than that, it's open season for making changes.

Handling Different Subversion Branches from Git

Of course, there's just the one local branch--master--and it's tracking just the one Subversion branch. That means any changes you make on master will always be sent back to that same Subversion branch. How do we send changes to a different branch? Just create a new local branch that tracks the remote-tracking branch you want to interact with. To create and switch to it with one command, run:

git checkout -b <new branch name> --track <remote branch name>

Now all commits made in your Git branch <new branch name> will eventually be sent to the Subversion branch <remote branch name>, and when you update from Subversion on that local branch, you'll get changes from that remote branch. Remember that you can use "git branch -r" to see all the remote branches that Git knows about. If you don't see the branch you want, then you'll need to use this command to refresh your remote-tracking branches with the latest Subversion changes, including new branches:

git svn fetch

Note: If there's a new branch to fetch, it can take a while, though not as long as the initial clone.

The one thing to keep in mind when branching in Git is to make sure to keep the history linear from Subversion's perspective. Don't branch from master and try to commit both the branch and master back to Subversion. Either merge them together, or commit one back, update the other to get it current, and then commit it, too. Again, I'm only interested in hearing about the details of the meltdown.

Those are the high points of cloning a Subversion repo and working in the clone. The next step is to be able to send commits back to Subversion and update the clone that you've made when someone else commits.

The Journey to Git, Part VII--Other Useful Stuff

2009-07-27T21:50:00.000-07:00

My previous Git posts were mostly a walkthrough of the basic workflow to get you up and running with Git fast. This post is less that and more a quick survey of other commands that are regularly used and/or useful. Previous posts aren't a prerequisite for this, but you need to at least have a repository with a few commits and branches in it to be able to run the commands and see what they do.

See What Changed

One of the most frequent commands is one ubiquitous to version control:

git diff

This command, by default, simply shows you what is different in your working tree from your index. In other words, it shows you what you've changed since the last commit but haven't staged yet. To see changes you've staged for commit, use:

git diff --staged

Of course, you can also use it to view the changes between any two arbitrary commits and/or branches:

git diff <commit|branch> <other commit|branch>

Note: Unless you want to see history in reverse, you always put the older commit first and the newer commit second.

And finally, you can see just the changes to a particular file or set of files by listing their names after the command and any options:

git diff file1 file2 ...

When you're using commands like this that refer to commits, it quickly gets old to look up their hashes, even when you can just copy/paste them. Fortunately, Git provides a concise vocabulary for specifying commits without using hashes. First, "HEAD" always refers to the "tip", or latest commit, of the current branch. You can also typically use the branch name to refer to the same commit, so we have at least one commit in each branch we can always refer to without knowing its hash. After that, when you know the name of any commit, you can use a caret to say "previous", so "HEAD^" means the commit before the latest commit on the current branch. Likewise, "master^" would refer to the commit before the latest commit on branch master. Carets stack, and each additional one signifies one more commit backward: "HEAD^^^^" is four commits before the latest commit on the current branch. This can also be expressed with "HEAD~4". Just use the tilde and a number to go back a specified number of commits. This is just the proverbial tip of the iceberg on specifying commits, but it's likely all you'll need for a great majority of what you'll do.

Given this new way of specifying commits, a command I use quite regularly is:

git diff HEAD^ HEAD

That is: show me what I did in the last commit. One final note: there are many diff viewing GUIs out there, but I'm not going to go into that much right now. If you've made it this far, you can probably manage setting one of them up on your own. I'll just point you at:

git help config

Search the output for "diff.external", and go from there. If you need more help, drop me a comment, and I'll see what I can do.

See History

The Command Line Way

Another often-used command that's common to VCSs is the log command:

git log

We've used this command in previous posts, but I'm going to add a few variations and a bit of detail to your toolbox here.

You might be accustomed to a log command that shows all the commits made on the current branch. Git does things slightly differently. The "git log" command shows all commits contained within the current branch. It's a subtle difference. When you merge a branch into another, not only does the merge commit show up in the destination branch, the commits from the branch that was merged appear as well. That's because all those commits are part of the state of that branch now. This can be slightly confusing to look at sometimes, but there's a handy option that helps you sort out where each commit came from when you need to:

git log --graph

The --graph option represents branches as lines to the left of the commits being shown. Each commit will have an asterisk next to it in one of the lines indicating which branch the commit was actually made on.

Note: The --graph option also changes the ordering scheme of the commits, potentially causing them to not appear in chronological order. I suppose this is supposed to make it easier to read the graph, but I find it distracting. Use the --date-order option to put them back in chronological order.

Another useful option lets you search commit messages and show only commits that match the search pattern:

git log --grep="some text"

Finally, sometimes it's handy to just see commits from certain times:

git log --since=yesterday

git log --until="15 Jul"

Note that none of these options are mutually exclusive. This is perfectly valid:

git log --graph --since="last week" --until="two days ago"

Note: I don't know what handles the date parsing in Git. I've seen it in the docs somewhere, but I can't figure out where. Whatever it is, it's very versatile.

The Fancy Way

Well, "git log" is great and all that, but what about when you really need to get down in the dirt and pick through the complete history of files? That's what gitk is for. It's similar to "git log" but packs much more detail into a screen. Gitk is one of the GUIs that comes with Git. In Cygwin and msysgit, it's installed along with the Git package, but on Linux, it's a separate package named--wait for it--"gitk". In any of them, the name of the command is also "gitk".

Run "gitk", and take a look around the interface. The top part of the screen is your list of commits, just like in "git log". Click a commit to select it. The bottom part is a diff showing what was changed in the currently-selected commit. In between the two is a control panel that, among other things, shows the SHA-1 of the selected commit, lets you search the selected commit for text, and lets you run rather powerful searches of all the commits appearing in the top pane.

In addition to all of that, "gitk" accepts many options that "git log" does, including some that lend themselves extremely well to the graphical representation. For one, you can see all commits from all branches with:

gitk --all

Another great feature is the ability to view any uncommon history between two branches you're thinking about merging:

gitk branch1...branch2

Run that way, it will show all commits from the latest commit on each of the branches back to the nearest common commit between the two: i.e. all the commits you're about to merge together.

Note: Gitk just runs off of the output from "git log", and it uses the --graph option when it does so. This means the commit ordering isn't necessarily chronological, like I explained above. Use "gitk --date-order" to get them back in order by date.

Hopping Around History

In previous posts, I introduced you to "git checkout" as a way to drop your changes to a file by getting the latest version of the file from the repository:

git checkout <path to file>

and as a way to move to a different branch:

git checkout <branch name>

In fact, "git checkout" can move you to any commit:

git checkout <commit>

This command pulls the state associated with the specified commit from the repository and makes it your working tree.

Note: Although you can use checkout to undo your changes to a file by getting that file from the repo, checking out a commit or branch is different. It isn't allowed to overwrite anything, and it does not perform a merge, so if something is in the way of what you're trying to check out, like a change in your working tree that would be lost by the checkout, Git will refuse to do it. There's two ways out of this situation: stash your changes and do the checkout, or use the -f option to "git checkout" to force the checkout, overwriting any changes.

The message from checking out an arbitrary commit brings up an interesting point: after you do it, you're no longer on a branch. You're on what Git calls a "detached HEAD". Interestingly, though, you can still do just about anything, including commit things. Since you're not on a branch, the commits naturally don't get applied to any branch, but they do happen. They're more or less in limbo, though, and you'll never see them again once you move back to a branch unless you do something to put them into a branch.

Since you can commit outside of a branch, it's rather important that you always ensure you're on a branch when you're working. Both "git status" and "git branch" show what branch you're on, if any. To move back onto a branch when you're not on one, just check out a branch again.

Looking Without Leaping

If all you want is to see what a file looked like at some time in the past, there's a much quicker way to do that than checking out the whole commit:

git show <commit>:<path to file>

Cool. Let's end on a short section. Stay tuned for still more Git goodness as I further explain how to interact with other Git repositories and with Subversion--a really cool feature!

Next stop, Subversion interaction.

Book Review: Agile Database Techniques

2009-07-22T14:54:00.000-07:00

I've decided to read a technical book every two weeks. You out there in tubeland will benefit by getting a book review every (roughly) two weeks. Here's the first, a book that I carried around with me for months meaning to read and finally decided I had to do it because there's a bunch more I want to read, too:

Agile Database Techniques

Effective Strategies for the Agile Software Developer

by Scott W. Ambler

The main theme of this book is the impedence mismatch between the traditional management of relational databases and increasingly agile software development, which somewhat mirrors that which exists between an RDBMS and an object-oriented software design. The basic premise is that DBAs still largely tend to define the entire database schema at the very beginning of a project and make it difficult to change, whereas developers are now largely accepting of the fact that software has to evolve instead of being fully designed up front. Some topics are covered because they relate directly to the main theme, and others are geared toward giving DBAs a basic understanding of and common vocabulary with modern software development and developers. Still others, such as the data normalization chapter, strive to do just the reverse for developers--give them a better understanding of the database side of the house. Overall, the author tries to bring DBAs and developers into a common ground where both are aware of the issues that the other must deal with and at the same time urge database professionals to adapt to Agile development, as the future is clearly one of an evolutionary approach to software.

Many of the chapters in this book are essential knowledge for an enterprise developer; however, it was published in 2003, and many concepts that were novel at that time are now taken as a matter of course, so you may already be familiar with much of it. For instance, if you have a good knowledge of Hibernate, you probably won't get much out of the chapters Object-Relational Impedance Mismatch (Chapter 7) or Mapping Objects to Relational Databases (Chapter 14), although they contain fairly important foundational knowledge. A good deal of the material in the book is like this, and it has quite a broad reach, dealing with topics from test-driven development to data normalization to UML.

The book is liberally sprinkled with real-world examples and practical advice. It comes through clearly that Ambler has been there and done that. There's also one chapter that covers a topic that hasn't gained much traction even today: database refactoring. The idea in this chapter is to attempt to loosely apply the idea of code refactoring--"a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior"--to the database to allow for more evolutionary database approaches even in the cases where multiple systems interact with one database. Ambler gives good examples that demonstrate why and when database refactorings are appropriate, and there is a catalog of established refactorings in the Appendix.

All things considered, I'd highly recommend this book for reading by junior developers. More senior ones should skim through it to be sure they're at least conversant on all the topics covered, as they are all very much relevant today.

The Journey to Git, Part VI--Rewriting History

2009-07-18T10:17:00.000-07:00

Welcome to my sixth (!!!) post on Git. I totally didn't expect this to go so long. In this post, we'll look at some methods Git gives you to change the history of your files. With centralized version control, there's a strong tendency to consider things irrevocable once committed. I'll show you that Git has no such constraints.

This post assumes you've read the previous ones in the series or are at least familiar enough with Git to use some of its basic commands. If you've been following along with the commands I've given in previous posts, this continues to build on the repository you've created as a result. If not, just create a repo and commit a file named "foo" to it with a line of text in it. Then add another line and commit again. This should give you enough to work with to see how the commands in this post work.

Changing History

Git isn't nearly so picky as SVN about changing or undoing things that you've done in version control. It supplies some very handy options for doing just that quickly and easily.

Note: While the commands are available, consider carefully what you're doing when you use them. Have the commits you're changing already been pushed or pulled out somewhere else? If so, will you or Git be able to resolve the differences if you start messing with the history? The easiest thing is to only change the history if the commit hasn't been propagated to another repository!

Amending Commits

The first command is a relatively simple one. Run:

git commit --amend -m "I decided to change this message"

Now "git log" will show that your most recent commit on the current branch has a different message. That's just the tip of the iceberg, though. You can amend a commit in any way. Modify the file foo again, stage the change, and do another amend commit:

echo "Another change to this file" >> foo
git add foo
git commit --amend -m "Foo needed to change again in this commit"

You can change your most recent commit in any way by simply staging some changes and using this command.

Undoing Commits with Reset

A more powerful history-altering command is "git reset". There are several options you can give this that do subtly different things. Let's look at the default form first. Run:

git reset HEAD^

The "HEAD^" means one commit before HEAD. This is a quick way of referring to commits rather than using "git log" every time to look up the hash. The carets stack, so "HEAD^^^^" would be four commits before HEAD, which can also be expressed as "HEAD~4".

Back to the reset: when you reset to some commit, you're making that commit the latest on the current branch. Any commits after it go away, but the changes they made aren't lost. All the changes from those dropped commits end up in your working tree, so "git status" will show you that foo is modified, and "git diff" will show that the change is the string that you most recently added to the file and committed.

Alternate forms of this, which we won't try here, are with a --hard and a --soft option (the default option is --mixed). With --soft, you get the same behavior as the default, but the changes end up in the index instead of your working tree. With --hard, you drop back to some commit, but all of your changes are lost. Use with caution. This is the Git equivalent of "rm -rf *". Note that the default target of reset is HEAD, meaning that "git reset --hard" will just erase all the work you've done since your last commit. This can be useful at times.

Totally Thrashing Commits with Rebase

Note: I believe you have to have your core.editor configuration option defined in order to use this since it relies on being able to edit text. See post two from this series for configuration stuff.

If you're only familiar with SVN, this one will blow your mind. Use "git log" to get the hash of your first commit, and run:

git rebase -i <hash>

In your text editor, you'll be presented with a list of commits and some basic instructions. The commits are all of the ones from the one after the commit you specified to the current commit--that's <selected commit> + 1 through HEAD, inclusive. By moving the commit lines around, you'll change the order of the commits in this branch. If you decide a commit is trash, delete the line, and when the rebase completes, the commit will be gone. Git does this by removing all of the commits you selected from the branch and then reapplying them in the order you specify.

In addition, you can specify a couple of other things. First, you can choose to amend any of the commits as they're reapplied by simply changing the "pick" before that commit to "e" or "edit". Remember when I said that "git commit --amend" only applies to the most recent commit? (I did say that, right? Back in post three, I think?) Well, that's technically true, but it doesn't mean you couldn't use rebase to go back ten commits to do an amend.

Finally, there's the squash option. "Squash" means compress two or more commits into one. Try this:

echo 1 >> foo
git commit -am "squashme"
echo 2 >> foo
git commit -am "squashme"
echo 3 >> foo
git commit -am "squashme"
git rebase -i HEAD^^^

Note: You'll notice that I used a new flag on commit that let me skip the "git add". Using -a with commit tells it to automatically add all modifications before committing. It only picks up changes to tracked files. The -a flag won't add any new files.

Now, let's squash the commits we made. Select "squash" or "s" for all three of the commits, save, and exit. You'll see the rebase happening, and at the end, you'll be prompted for a new commit message.

This interactive rebase (i=interactive) really points to how Git differs from centralized version control with respect to its attitude toward history. In Git, history is completely mutable until it gets too complicated to control by being replicated to other repositories. As long as it only exists in your own repo, you can change practically anything about it. Git views commits as part of the development process, and therefore recognizes them as things that might need to be cleaned up a bit after the fact. In, for instance, SVN, on the other hand, the general attitude is that once it's in the repository, it's set in adamantium.

Note: I believe that in newer versions of Git, the pop command has been removed in favor of apply, which also existed in older versions, but had different behavior. Check "git help stash" to see the commands available.

In the next--and hopefully last--post that deals with using Git locally, I'll show you how to navigate around your repository, and find out things you want to know about your files and history.

Vote for this article on DZone.

The Journey to Git, Part V--Merging

2009-07-18T10:16:00.000-07:00

This post is one of a series on Git. Previously I posted on branching. When you create a branch, you diverge two lines of development. You need a way to join them back up later on, and that's what this post covers.

Before starting this, make sure you've read my other Git posts leading up to it or are comfortable working with branches in Git. This post also assumes you've been following along with the commands in the other posts. If not, you should create a repository with master and put a file named "foo" with a couple lines of text in branch master. I highly recommend that you follow along with the commands so you can see Git working.

Merging Branches

Create a branch from master and move onto it with:

git checkout -b mergeme

Create a file named "baz" with some text in it, and commit it on this new branch:

echo "File on branch mergeme" >> baz
git add baz
git commit -m "Added baz on branch mergeme"

Suppose that's all the work there is to do on mergeme, and we want to merge the final result into our master branch. Just switch to master and merge the branch into it:

git checkout master
git merge mergeme

You'll see output from the merge like:

Updating 645663b..bb64c2c
Fast forward
 baz |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 baz

The first line indicates the two commits participating in the merge with the starting location first and the merge target second. The next line says "Fast forward" because Git didn't actually have to merge anything in order to make this happen. Since no commits happened on master between the time you created mergeme and the time you merged it back in, Git was able to simply "fast forward" master, meaning it just took the changes in the branch and replayed them on master. Had you made another commit on master before merging mergeme back, Git would have actually had to merge the commits together to create a new commit, which brings us to our next topic: any time an actual merge is performed, you have a chance of entering the lovely land of merge conflicts.

Resolving Conflicts

Merge conflicts aren't much different in Git than in any other version control. Let's create one by appending a different line to foo in each of the branches and attempting to merge them:

git checkout mergeme
echo "Conflicting line in mergeme" >> foo
git commit -am "Conflict in mergeme"
git checkout master
echo "Conflicting line in master" >> foo
git commit -am "Conflict in master"
git merge mergeme

If it broke as expected, the merge will have produced output like:

Auto-merging foo
CONFLICT (content): Merge conflict in foo
Automatic merge failed; fix conflicts and then commit the result.

When a merge is successful, the result is automatically committed. However, when there's a conflict, the results of the merge stay in your index and/or working tree. Run "git status" and examine the output. It notifies you of files needing to be merged manually, both at the top of the status output and by marking them as "unmerged". You simply need to resolve the conflicts either manually or with your tool of choice, and make sure the file gets added, then commit the final result yourself. If you've set up your own merge tool--see post two in this series--you can fix the conflict with:

git mergetool

Otherwise, just adjust the content of foo to remove the conflict markers, and add it:

git add foo

Either way, all changes should be staged, and you can now:

git commit -m "I merged this myself"

This merge resulted in a new commit in master because it's a real merge that entailed modifications, whereas the fast-forward we did before essentially just copied the commit from one branch to another.

That covers the basics of merging. I have a feeling I missed something, so this post may change, but for now, let's move on to the next post, where I'll show you how to rewrite history with Git. It's pretty cool: Part VI--Rewriting History

Vote for this article on DZone.

The Journey to Git, Part IV--Branching

2009-07-18T10:15:00.000-07:00

Git post 4: Branching. Before you read this post, you should be familiar enough with Git to be comfortable creating a repository and adding and committing files to it. The previous posts in this series will bring you up to that point if you aren't.

Note: This post assumes that you followed along with the commands given in the previous post and have a version-controlled directory with a couple of commits in it. If you don't have that, then create a new git repository in an empty directory, and create a file named "foo" with one line of text in it and commit it. That will get you to where you need to be to start this walkthrough.

I'm going to assume that you're already familiar with the concept of a branch in a VCS. What you need to know is that Git branches are first-class citizens. They're very lightweight and easy to create and work with. With Git, it's not a question of if you'll branch, but of how many branches you'll have going at once.

Creating Branches

Let's create a branch for some concurrent development:

git branch mybranch

That produces no output if it's successful, but it creates a new branch named "mybranch" starting from your current location: the most recent commit of your current branch. View your current branches with:

git branch

It should show the following, indicating you have two branches and are currently on branch master:

* master
  mybranch

Now switch to the branch you just created:

git checkout mybranch

Note: You can both create and move to a branch at the same time with: "git checkout -b mybranch"

In a previous post, you used "git checkout" to pick out a specific commit and make that commit your working tree. It does essentially the same thing when you give it a branch name as an argument. In another post, I'll cover checkout a bit more thoroughly. For now, let's create a new file on this branch:

echo "Branch file" >> bar
git add bar
git commit -m "Added bar on mybranch"

Now "git status" should show that you're on branch mybranch and have nothing to commit, while "git log" will show all three of the commits you've made so far. Move back to master with:

git checkout master

Now bar is gone because it only exists on mybranch, and you'll see that "git log" doesn't show the commit that added it. That's because it shows the commit history of the current branch.

Renaming Branches

This will be a short section:

git branch -m mybranch myotherbranch

That's it. The branch is renamed.

Deleting Branches

Deleting branches is almost as simple. Try this:

git branch -d myotherbranch

It should give you a message like:

error: The branch 'myotherbranch' is not an ancestor of your current HEAD.
If you are sure you want to delete it, run 'git branch -D myotherbranch'.

The keyword "HEAD" refers to the most recent commit of the branch you've checked out. What it's telling you is that you have commits in myotherbranch that aren't in your current branch. If that weren't the case, the delete would have succeeded. Git is very flexible, but it tries to protect you from losing things accidentally. Do what the error said and use the flag that forces a delete:

git branch -D myotherbranch

You'll see a confirmation message. That branch--along with the changes on it--are gone. Make a habit of using the -d form, and you're less likely to accidentally lose things by deleting branches that have changes that don't exist elsewhere.

That's really all there is to branching in Git. Like I said, it's very easy to work with branches in Git. Stick around for more branch goodness in the form of merging in my next post: Part V--Merging

Vote for this article on DZone.

The Journey to Git, Part III—The Basics

2009-07-18T10:14:00.000-07:00

This post: the most basic commands for interacting with a Git repository. By the end of this post, you should be able to use Git to track basic history for a project. Once again, I strongly encourage you to follow along with the commands in your own environment to help you learn.

Creating a Repository

Make a directory named firstgitproject and change to it. Run:

git init

You'll see a message like, "Initialized empty Git repository in <path> to project>/firstgitproject/.git", and if you look in your project directory, you'll see that there is, indeed, a .git folder. This folder contains the repository you just created along with its configuration.

Status of a New Repo

Let's see what Git has to say about your brand new project with:

git status

You should see three things: that you're "On branch master", that this is the "Initial commit", and that you have "nothing to commit". The "master" branch is always the branch you start off in. In ways, it's similar to SVN's "trunk", but unlike in SVN, there's nothing special about it. You can rename or delete master with no problem. It's just a branch like any other you might create. The "Initial commit" is a little mysterious in Git. It signifies that you have no history to the project yet. Many Git commands will exit with errors when run against this "Initial commit"--i.e. until you make your first commit to the repo, so let's work on that.

Adding Files/The First Commit

Create File--Working Tree

The directory that you're in, where the .git directory is and where all your files for this project would ultimately go, is what Git refers to as your working tree. It's where you do all your work. Let's add a file to your working tree:

echo "Hello World" > foo

Run "git status" again. Now you have an "untracked" file named foo. Untracked means what it sounds like: Git isn't tracking this file. It exists only in your working tree. Let's change that.

Stage Change--Index

Tell Git to track your newly created file with:

git add foo

Another "git status" now shows foo under "Changes to be committed". What you just did, in Gitspeak, is called "staging a change". You took something that has changed in your working tree (a "change") and told Git to include it in the next commit ("staged it") using the add command. Git has a name for this area where you stage your commits to: the index. I don't know the rationale behind the naming, but when you see the documentation refer to the "index", this is what it's talking about.

Commit Change--Repository

The final step is to commit, which takes all the changes in your index and moves them into the repository. You should be familiar with the concept of a repository. It's where the full history of your project is kept. The general workflow, which we just simulated, is this: you have a working tree and index that are in sync with the repository--you've changed no files and staged no changes. You make changes, and now you have a dirty working tree. You stage some or all of the changes to the index, and now you have a clean working tree and dirty index. Then you commit the staged changes to the repository, and you're back to a clean state. Repeat.

So let's finish the cycle:

git commit -m "My first commit"

Note: If you've set the core.editor configuration option, you can omit the "-m <message>" and your editor will be opened, showing you the complete status message. Just type your commit message, save, and exit to complete the commit

The response of this command should look something like:

[master (root-commit)]: created b1468b4: "My first commit"
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 foo

This indicates several things that I'm going to explain only briefly here. First, you made this commit on branch "master". This is the root-commit, meaning the very first in the repository--the one that went on top of the "Initial commit". A commit identifiable by "b1468b4" was created with the given message. In this commit, one file was changed, a single line was inserted, and no lines were deleted. Finally, this commit included the creation of a new file named "foo" with mode 100644 (that's the Linux file mode that indicates file permissions for you Windows guys).

At this point, some people may be surprised at the fact that we just made a commit without having any kind of Git server set up. Recall that Git is a Distributed VCS. You have an entire copy of the repository, and it lives in that .git directory, remember? So all that's involved in a commit is some local file operations. There's no need for any kind of client/server setup. Now back to our regularly scheduled programming.

Modifying Files Under Version Control

Make Change--Working Tree

The file "foo" is now under version control. Let's make a change to it:

echo "Line 2" >> foo

Now a "git status" will indicate that foo is "Changed but not updated". This message is a little ambiguous. What it means is that the file is under version control, but the copy of the file in your working tree differs from that in the repository, and you haven't staged this change yet. It's time to look at a massively useful command, which you're surely familiar with from other VCSs:

git diff

This is probably the most common use of diff, though there are a host of other options available for it. I'll cover some of those later on.

You've now seen all three parts of the "git status" output. To recap, the output contains 1) the untracked files which aren't under version control, 2) files under version control that you've made changes to, and 3) changes--either new files or modifications to version controlled files--that you've staged and will be included in the next commit.

You'll also see in this latest status output that it prompts you with a couple of ways to deal with this file. You can either "add" it--we'll do this in a moment--or "checkout" it. In SVN, you're used to "checkout" being a very rare command used only for the initial retrieval of a project from a repository. In Git, it's a much more common command which means, roughly, "get X from the repository and put it in my working tree". If you were to "git checkout foo", then foo would be replaced by the version in the repository, removing your most recent changes. In this way, it acts like SVN's revert command. Let's not do that now.

Stage Change--Index

Instead, stage your change with:

git add foo

This is something that regularly trips up SVN users trying out Git. In SVN, you only add newly created files. In Git, you "add" every change. It's actually a more consistent approach and very logical when you get used to the idea. Remember that "add" always stages a change--moves it into the index--and it's the index that gets committed. Another "git status" at this point will show you a similar output to before, when you had staged the newly created file, showing that a new file and a modified file are both just considered "changes" that have to be incorporated.

Commit Change--Repository

Go ahead and commit now with:

git commit -m "My second commit"

Take note of the difference between the output from this commit and the previous one. It's somewhat briefer since this isn't the root-commit and there wasn't a new file created.

Seeing Your History

Now look at your commit history with:

git log

This brings up a really handy feature of Git. If the output of any Git command exceeds what will fit on one screen, Git automatically pipes the output through less, making it easily browsable. Once your history grows beyond a few commits, you'll see this behavior regularly with "git log".

In the commit log, you'll see for each commit 1) the unique identifier of the commit--a 40-hex-digit SHA-1 hash, 2) the author of the commit, 3) the timestamp of the commit, and 4) the commit message entered for the commit.

Removing Files from Version Control

Now for one more, trivial command:

git rm foo
git commit -m "I removed foo"

That's pretty self-explanatory, I think. It's what you do to delete files. Go ahead and add foo back, so that we can use it some more:

echo "new foo" >> foo
git add foo
git commit -m "Added foo back"

View Previous Versions

Establishing a history of things isn't very useful if you can't look back through the history. Here's the basic form of two commands to let you get a feel for a simple history. Later on, I'll do a whole post on navigating around a repository and seeing what's in it.

Show a Previous Version

Use "git log" to get the hash of a previous commit, and run:

git show <hash>:foo

That will show you the content of the file foo at that commit. Now use "git log" to pick two different commits, and run:

git diff <older hash> <newer hash>

You'll see the differences between the two versions in patch format. Added lines have a '+' in front, and deleted lines have a '-'.

I'm going to break this post here, though there's lots more to cover. By now, however, you know all you need in order to keep a simple, forward-moving history of any personal project you happen to be running. It's also handy for versioning configuration files if you run, for instance, a Squid proxy server or other local service with complex, text-based configuration. The next couple of posts will primarily cover branching and merging, where Git really shines in comparison to centralized version control.

First, branching: Part IV--Branching

Vote for this article on DZone.

The Journey to Git, Part II—Git Started

2009-07-13T23:13:00.000-07:00

This second post in my Git series will just cover setting up Git so that I can keep these things fairly short and well-organized. From here on, everything is very hands-on. If you're serious about learning Git, I highly recommend you follow along with all of the steps to reinforce what you're reading. This is all going to be a command line walkthrough, though Git ships with some nice GUI tools as well. Once you're familiar with the command line and the concepts, the GUI tools should be easy to figure out.

Installing

First, obviously, install Git. On Linux, it's a simple package installation. On Windows, you can use either msysgit or install the Git package in Cygwin. I've used msysgit only in passing. Most of my experience is evenly split between Cygwin and Ubuntu. The only other thing I'll say here is you do not want msysgit and Cygwin installed at the same time (which is why I didn't use msysgit much). The two are not compatible, and you'll likely see some odd behavior in Cygwin if you do it (like scp thinking your home directory is "c:\program file\msysgit"). If you need further details on installing Git, email me, and I'll try to help.

If Git is properly installed, then you should be able to run:

git --version

Configuring

There's some configuration we can get out of the way up front. This is all optional, in fact, but at least the first one is certainly a good idea. Your configuration tool in Git is "git config". By default, this command updates a repository's configuration, but with the "--global" flag, it sets options for your user system-wide, so we can set up some preferences before we even have a repo. For more info about the command and a full list of configuration options, see the man page for "git config".

Identify Yourself

Let Git know who you are with:

git config --global user.name "your name"
git config --global user.email "your email"

This name and email are included in the information about every commit that you make.

Pick an Editor

There are a few things that Git needs to open a text editor for, like typing commit messages if you don't want to type them in-line. Give it the path to your preferred editor with:

git config --global core.editor <path>

Pick a Merge Tool

For some reason, I've never gotten into graphical merge tools, preferring instead to just view the raw conflict markers, but I know some people wouldn't even glance at a VCS that didn't offer the option of merge GUIs. Git knows about several Linux merge tools, and if you want to use one of those, you can just use:

git config --global merge.tool <tool>

Built in merge tools are listed in the man page. If you want to use something other than the built-ins, which you probably will if you're on Windows, it's a bit more work. Assuming you're using WinMerge:

git config --global merge.tool winmerge
git config --global mergetool.winmerge.cmd "WinMergeU \$MERGED"

If WinMerge isn't on your path, you should be able to specify the path to it with:

git config --global mergetool.winmerge.path <path>

However I couldn't get that to work. It's fine if you just leave off the path option and add it to your system path.

For Windows

If you're using one of the Windows Git flavors, you'll very likely want to set these options to handle line terminators more gracefully:

git config --global core.autocrlf true
git config --global core.safecrlf true

And if you're using msysgit, you may need:

git config --global core.fileMode false

I had trouble with it that this option fixed.

There are tons more configuration options documented in the man page, but these should get you started. In the next post, we'll finally get to basic, day-to-day commands that will let you start using Git productively: Part III--The Basics

Vote for this article on DZone.

The Journey to Git, Part I—Distributed vs Central VCS

2009-07-13T22:53:00.000-07:00

This one has been a long time coming. I've been using Git as a client to Subversion at work now for a number of months. Over the weekend, I attended a session on Git at the NFJS conference, and it served to solidify some of the concepts in my mind. I think I'm ready to do one or more posts on the subject now.

This first post is going to be more an examination of distributed vs. centralized VCS than anything specific to Git itself. The remainder of the posts in this series of currently unknown length will be a quick start tutorial to get you up and running, teaching you how to use Git at a very basic level with just a single, local Git repository. When I was trying to get going with Git, I couldn't find a good reference online to get me up and running in the way I needed, so I'm going to try to make this as complete as possible. I assume zero Git knowledge here, but I do assume you're familiar with version control in general.

Why Git?

First, I'm going to examine why we even need to care about a new VCS. What does Git have that SVN doesn't? That really comes down to the fact that they're two different beasts. It's not so much Git vs. SVN, but distributed VCS (DVCS) vs centralized VCS.

Centralized VCS

With centralized version control, a development team is locked in in terms of workflow. You have a local copy of the project (or more than one). You can make changes to your heart's content, but everything is strictly local until and unless you commit back to the repository. Then whatever you did is visible to everyone. There's no in-between here, no compromise. I actually find it quite remarkable that developers the world over have been content with this model for so long. There are a few very immediate problems that come up.

Problems with Central Control

#1: Collaboration

Scenario: You're working on adding a new feature to an application and you want another developer to give you a hand with part of it or look over some code you think is questionable, or maybe he's just working on something else that interacts with your piece. Your code needs to be available for the other guy to work with. With a centralized VCS, your only options are to 1) commit your code, making it visible to everyone else, or 2) share it some other way outside of the VCS. Option 2) is just messy, if workable. Option 1) would probably be preferred for the most part. The problem here is that your code could be in very early development, possibly even broken. We all know there's a horrible taboo on committing broken code, and rightly so with central control. So you're stuck with not being able to share or with having to stop work to make things non-broken to the point that you can safely commit before you can collaborate on a piece of code.

#2: Releases

This problem is significantly more insidious. Any time you commit something to a centralized VCS, you're essentially affirming that this feature will absolutely be included in the next release. This assumes, of course, that you're not doing work on your own branch specific either to your feature or yourself--a whole other worm can with centralized version control. As your release process becomes more streamlined and automated (that's happening, right?) and your release cycles get shorter because you're aiming to be a more agile--that's agile rather than necessarily Agile--team (that's happening, too, right?), it becomes more important to not schedule features for a release, but to schedule a release for a date and put in whatever features are finished on that date. This means it's imperative that unfinished features never be committed to your mainline development branch, like trunk, since you can't know exactly when a feature will be ready for release. Instead, features should be developed in isolation and only moved into the "trunk" branch when completely finished. This leads back to the branch-per-feature idea, which central VCS in general and SVN in specific just aren't designed for.

#3: History

This one's simple and may be less compelling to some. It's a big hangup for me. A VCS is all about keeping track of history, right? Well, who ever said history was perfect? Only a centralized VCS tries to enforce that by demanding that every commit be 100% bug-free since everyone is going to see it. This leads to larger, less frequent commits, potentially as much as days apart. Days! Why have we come to accept this as the norm? I say version control should be no more than an hour behind you at worst. Small, incremental changes are better. VCS should be a coarse-grained undo system.

Distributed VCS

How does a DVCS answer these concerns? Laying out the problems took quite a bit of space, but interestingly, they're all solved by this fundamental difference: in a DVCS, you not only have your own working copy, you in fact have your own copy of the entire repository, or at least the pieces you care about. You can change anything about it in any way, and nothing leaks out to anyone else unless 1) you push it out or 2) someone else pulls it from you intentionally. This solves all three of my problems since nobody, including a release, will see what you've done until you're ready for them to, but at the same time, your changes are available to anyone who's interested.

Now that you know why a DVCS is a good idea, let's get back to Git itself. There seem to be three mainstream DVCSs out there today: Git, Mercurial, and Bazaar. Of the three, Git seems to be most widely used and gaining traction. That's not necessarily a reason to use it, but Git is what I've used, so it's what I'm writing about.

See the next post to begin a walkthrough tutorial of Git: Part II--Git Started.

Vote for this article on DZone.

NFJS the Beginning

2009-07-12T21:34:00.000-07:00

I just got home from the Austin, TX No Fluff Just Stuff conference. Wow! It's the first one I've been to, and it was great. For my reference and yours, I'm going to put some notes here of things that bear remembering. I expect I'll make several posts over the next few days (or more) on topics covered at the conference. Today:

Open Source Debugging Tools

One of the really good sessions I attended was on a variety of open source tools useful for debugging java apps. It's getting to the point, or maybe it's already there, where there's almost no benefit to shelling out the cash for a commercial debugging/profiling application like YourKit. Think “tools” in a loose sense here, because some of these things are just handy CLI commands that make your debugging efforts more efficient. If you're too lazy or don't have time to ready the whole thing, the "must-see"s for me in this list were OQL (in jhat or Eclipse Memory Analyzer--I don't think VisualVM supports it yet), omniscient debuggers, and BTrace.

I believe all the tools I mention are available for Windows, Linux, or Mac unless otherwise stated. Certainly the Java-based ones are. It started off with tools that aren't actually Java-oriented, but instaed are for network debugging:

cURL – absolutely raw HTTP interaction. This guy is great for debugging any of the many protocols that use HTTP: normal web requests, AJAX, SOAP, web services, etc. You can specify any request method, parameters, body, headers, etc. Also has modes for HTTPS, FTP(S), SCP, and more.
Tcpdump – watch raw network traffic at the packet level. This tool shows some or all of the content of packets passing through the network interface of your computer. It's highly configurable to capture only exactly what you want.
Wireshark – GUI for viewing tcpdump output, formerly known as Ethereal. Wireshark also includes tcpdump and can be used to do the captures. Often, though, you're working in a headless environment, in which case you'd need to use tcpdump and ship the dump file back to where you can analyze it with the excellent Wireshark interface.

Now to the Java tools. A “**” indicates that a tool is distributed with the JDK since version 1.5. I got tired of typing it over and over:

Jmeter – Apache Jmeter has come a long way. It's primarily a load testing tool that can run against a number of different types of servers that one typically encounters in Java enterprise development. The interface isn't entirely intuitive, but don't give up too quickly. It is extremely flexible and powerful.
Jps** – like ps in Linux, but shows Java-specific details about all running JVM processes.
Jstat** – shows JVM statistics for a particular JVM. This shows mostly memory- and garbage-collection-related stats.
Jconsole**/VisualVM – graphical tool that shows tons of statistics about a JVM as well as JMX Mbeans. They can also initiate and analyze heap dumps. Jconsole is being deprecated in favor of VisualVM. While VisualVM is actually included with the JDK, it's recommended to download it separately since development is ongoing, and the version from the website will likely always be way ahead of the version in your JDK.
Jstatd** – a daemon that allows you to connect jps, jstat, and jconsole/VisualVM to remote machines. I.e. run jstatd on the remote machine, and then you can connect one of the other tools to it.
Jstack** – shows stacktraces of all threads in a running JVM.
Jmap** – displays/dumps content of the Java heap. A heap dump can be very large and expensive, but invaluable in diagnosing performance problems and OOMEs. Dumps are created in a standard format that can be read by a number of tools.
Jhat** – analyze and display—actually run an HTTP server that serves the results of—a heap dump. Jhat is quick and effective (given enough of its own heap space to run in!), but since it exposes everything as HTML, it's somewhat limited in comparison to Jconsole/VisualVM or another tool that can parse heap dumps.
OQL – not a tool, per se, but I only learned about it this weekend, and it's awesome! Object Query Language is a language for letting you query a heap dump to find out virtually anything you could possibly want to know in one, neat result set! Simple example: “select z from java.lang.String z where z.count > 50” – applied to a heap dump will display all String instances with more than 50 characters. I believe a number of tools, including jhat, support OQL, but I haven't researched this enough yet.
Jinfo – display or change JVM options while a JVM is running. For instance, turn on HeapDumpOnOutOfMemoryError on every important JVM you're currently running because it could help you immensely in diagnosing crashes.
Javap – class file disassembly. Given any java class file, show a human-readable representation of the bytecode. If the class was compiled without debug information, it will be somewhat less readable.
Eclipse Memory Analyzer – Eclipse plugin that can load (but not initiate) heap dumps and run OQL queries against them and has some quite friendly and useful visual representations of the dump.
BTrace – a utility that lets you easily inject your own code into a running JVM by dynamically instrumenting bytecode. The code is quite simple. You just have to write a static method and put an annotation on it telling it when to run. Note: the code you can put in a BTrace class is extremely limited. It seems to be mainly intended for creating more verbosity in an application when things aren't working quite right and your logging isn't telling you what you need to know.
“Omniscient” debuggers – We didn't have time to get into these, but they look crazy powerful. The idea with these is that the debugger knows everything and will let you do anything, including stepping backward through a program, to figure out where a bug came from. They do all this by recording every single thing that happens in the JVM. Thus while they're crazy powerful, they're also crazy slow, but still... A couple that were mentioned: ODB and TOD.

Finally, a few more non-java tools that deal with general OS stuff:

fs_usage (Mac), inotifywait, inotifywatch, lsof (Linux) – tools for watching changes to files and/or seeing what processes are changing files.
Process Monitor (Windows) – lets you see what processes are changing files and much, much more for Windows. Not open source, but free. This tool is like Task Manager on steroids.
FileMon (Windows) – close cousin of Process Monitor that shows lots more file-related information in Windows. Also not open source, but free.

How To Aggregate Downstream Test Results in Hudson

2009-07-08T18:21:00.000-07:00

I must've googled a dozen permutations of this post's title looking for the key to my problem of... well, what it says. How on earth do I get Hudson (a top-notch CI server) to "Aggregate Downstream Test Results"??? The little help icon, which is typically very useful, provided a handy description of the feature, but didn't say anything about how to use it. My searches found several posts from other people having the same problem, but little in the way of resolution. I'm not sure if there's just not many people wanting to do this or if the "how" was just very obvious to everyone but me, but one way or the other, I finally figured it out. In retrospect, it does seem a bit obvious.

The key to getting Hudson to track test results from downstream builds is that all of the builds, upstream and down, must have a common, fingerprinted artifact. Any old artifact will do...almost. More on that later. I just had my first job--let's call it Foo--create a file with the date and time in it called "starttime".

1) Have Foo archive this file as an artifact, and either enter the file in the list of artifacts to fingerprint, or turn on fingerprinting for all artifacts.

2) In the downstream job, Bar, do exactly the same as in step one for Foo: archive and fingerprint the same file. Of course this implies that Bar has actually gotten the file in question somehow. I haven't worked out how to do this through Hudson other than by using wget to retrieve the archived one from the last build of Foo: something like "wget http://localhost:8080/job/Foo/lastBuild/artifact/starttime".

3) Of course, you'll need to have some test output being recorded by at least Bar, if not both jobs, since the whole point of this is to see aggregated test results.

4) You can always see the very latest test results at http://localhost:8080/job/Foo/lastSuccessfulBuild/aggregatedTestReport. Don't use "lastBuild" or the page will be unavailable when Foo is running or broken. When you have a lot of downstream tests, you can enable auto-refresh on this page and see the results fill in as jobs complete--very cool, especially when you have a nice test cluster with lots of downstream jobs.

5) It's handy to make Bar triggered by a build of Foo, but it's by no means required. From now on, any builds of Foo and Bar that share an identical artifact (checked by md5 checksum) are linked together, and the link appears in several places, one of which is the aggregated test results.

I emphasized the word "any" there for the same reason I said you can use almost any artifact. Remember it's the md5sum of the artifact in question that is used to determine which builds of which jobs are linked together. Say you start builds Foo #4, #5, and #6, and all create or use the same "starttime" file for some reason--maybe you're using just the date instead of date + time. Then Bar #4, #5, and #6 also all retrieve, archive, and fingerprint the same file. Since the checksum of the artifact is common in all those builds, they'll all be linked together, and Hudson can't really tell them apart. I'm not sure what this does for aggregated test results, but in other places, like where upstream and downstream builds are listed, instead of showing that Foo #4 led to Bar #4, it'll show Foo #4 led to Bar #4-#6.

So there ya go. Let 'er rip. I'm happily chugging away now with a 5-node Hudson cluster churning out test results like there's no tomorrow.

Using Oracle's OCI Driver With JDBC

2009-04-02T09:24:00.000-07:00

I just spent a morning learning how to use the Oracle OCI driver instead of the thin driver, and it's worth recording for future reference. I think it used to be more complicated, and you had to actually install some Oracle app or other, but as of version 10.1.2 or so, the process is somewhat simpler and just requires a couple of downloads. If you've always avoided using OCI to connect from Java or just never had a reason to do it before, here's what you need to do: 1) Download the Oracle "Instant Client", which lets you use OCI. I don't know exactly what these terms really mean in Oracle-speak. Just go to this download page, select your platform, then find your database version and download the "Instant Client Package - Basic" and "Instant Client Package - JDBC Supplement" packages. (You can use the Basic Lite package in place of the first one if you only need English/western i18n stuff.) You'll be prompted to log in to or register for OTN, which is a free registration. 2) Make sure you use the SAME DATABASE VERSION of the JDBC thin driver. If you don't already have this JAR, it comes in the Instant Client zip file (named ojdbc14.jar). You can also get it from this page if you really feel the need. It's important that the versions match, because the thin client uses native calls to access the OCI stuff, and the file names are hardcoded (I suspect) into the thin driver. So if you use version 10.x of the JDBC driver, but version 11 of the OCI driver, you'll get an error like "java.lang.UnsatisfiedLinkError: no ocijdbc10 in java.library.path".

3) Extract the two Instant Client downloads into the same directory somewhere.

4) Create your tnsnames.ora file somewhere (if you don't already have one). This file describes how to connect to one or more Oracle databases to any Oracle software on your computer, such as the Instant Client. It's widely documented on the web, like on this page. If you have a RAC and don't know much about this file (like me before this morning), then someone else probably set up the cluster. Get a tnsnames.ora entry from them and just paste it into a text file somewhere. I just put the tnsnames.ora in the same directory where I extracted the Instant Client.

5) Add/edit two environment variables. First, add the directory from 3) to your library path. In Windows, this means adding the directory to your PATH. For Linux:

export LD_LIBRARY_PATH=<directory>

Then add a TNS_ADMIN variable that points to the directory that contains tnsnames.ora (the same as the library directory if you put it with the rest of the extracted files).

6) Use JDBC to connect like normal with these settings (and a username + password, of course, if applicable):

driverClass=oracle.jdbc.driver.OracleDriver
connectionUrl=jdbc:oracle:oci:@<database alias>

where <database alias> is the alias specified in tnsnames.ora. It's the left hand side of the initial assignment in a TNS entry--"FITZGERALD.SALES" in this example:

FITZGERALD.SALES =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = server1)(PORT = 1521))
    (CONNECT_DATA =
      (SID = ORCL)
    )
  )

That's it! Section 7.4.3 of the Oracle JDBC Developer's Guide has a good rundown of different ways you can use JDBC to connect with OCI once you get it installed. I don't claim that this is the best way to do any of this. I just know it works.

Put Your Screen Saver to Work

2009-03-24T08:28:00.000-07:00

Ever intend to start your computer working on a big job when you walk away so it'll be done when you come back an hour later? Ever walk away and forget to start it? That was happening to me a lot, so I wrote a screen saver that runs stuff while it's active, thereby automatically making use of my computer's idle time. I'm posting it here in case any of you would like to use it.

Since there doesn't seem to be an attachment mechanism, it's embedded in the image to the left. Just follow the instructions on the image. I got this simple method of embedding the file in the image from here: http://www.tricksystem.com/2008/12/secretly-hide-any-file-inside-jpg-image.html. You should get a file out of it named "Jobs@Home.scr". That's the screen saver. Just put it in c:\windows\system32, and it'll be selectable as a screen saver. MD5 checksum if you want to check it:

$ cat Jobs@Home.scr.md5
1f190b9e2db877fe07d683f551d621a3 *Jobs@Home.scr

How it works:

First, it's Windows-only. Sorry Linux guys. I wrote and am using it on Windows XP Home and Professional. Next, it's quite primitive. There is no configuration of the screen saver itself. It's hard-coded to read from a file named "c:\ssavejobs". This should be a plain text file with a command per line. The screen saver reads the lines from the file and executes each one in sequence (not parallel). When it finishes executing one line, it removes it from the file and begins the next one, so the file acts as a FIFO queue. Each line is executed by running the following command in a new process:

c:\windows\system32\cmd.exe /c <line from file>

Note that the ssavejobs file is *not* a .bat file. It is simply a series of one-line commands, each of which is executed as an independent process.

Example ssavejobs:

copy c:\test.txt c:\test2.txt
del c:\test.txt
move c:\test2.txt c:\test.txt
c:\foo\bar\rip.bat d: c:\rippedMusic
c:\foo\bar\encode.bat c:\rippedMusic\*

With the above example, the screen saver, when it comes on, would copy a file, delete a file, move the file back where it was (just trivial stuff to give you an idea of how it works), then rip music from your d: drive to a directory on your c: drive, and finally encode all the ripped music (assuming you have a rip.bat and encode.bat file that will do that for you). Keep in mind that unless you expect to be away from your computer for long enough for all the jobs to execute, you should make each job as short as possible. That way, when you do come back, either the current job will finish quickly, or you can cancel the current job without losing a ton of work. I.e in the above example, it would be better to rip and encode each track (or whatever) individually than doing two, large batch jobs.

Caveats:

The big one is that the screen saver can only update the ssavejobs file when it's on. That means if you come back to your computer and kill the saver while it's in the middle of a job, the job will continue (it's a separate process), but that same job will also remain at the top of the ssavejobs file. If this happens, you should either kill the process and let it restart later or let the job finish and manually remove the top line from ssavejobs. I might try to fix this if it becomes too big a headache.

Another problem is that the screen saver starts its background work when it displays in the "Display Properties" screen saver tab--you know, the tiny preview of it that you see in the picture of the monitor. So you might inadvertently start it by going to the screen saver dialog. It also starts, obviously, when you use the Preview button.

A potential deficiency is that the stdout of the processes being executed isn't captured anywhere, so you have no logs of what happened. You can only look at whatever outputs you were expecting to see if thing went as expected. I'll probably change this to print stdout/stderr to a log file if I can figure out how.

Finally, the dots (the screen saver itself) are not my doing. They were in the skeleton code that I downloaded to explain how to write a screen saver. They're more exciting than a blank screen, so I've left them on for now.

Generic Non-Erasure?

2009-02-19T07:56:00.000-08:00

I was coding away the other day and got this error:

Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String

on code that looks in essence like this:

public class Example {
    public static void main(String[] args) {
        List<String> stringList = legacyGetList();
        System.out.println("First element: " + stringList.get(0));
    }

    static List legacyGetList() {
        List untypedList = new ArrayList();
        untypedList.add(new Integer(5));
        return untypedList;
    }
}

It happens on the fourth line, the System.out.println. What's the deal? It can't be related to the type parameter, can it? Generic types are erased at runtime, right? Well, it turns out that if you change List<String> to List<Object> or just List, then it works. Why?

Because of a compiler optimization. String concatenation compiles into StringBuilder invocations (at least it has since StringBuilder was introduced). StringBuilder has several overloaded append(...) methods--one for each primitive type, one for java.lang.String, and one for java.lang.Object--for convenience and, I guess, performance. So ask yourself: when is a method selected for execution from a list of overloads? If you answered "at compile-time", then you're right. If you answered the other way, prove it to yourself:

public class AnotherExample {
    public static void main(String[] args) {
        foo((String) null);
        foo((Object) null);
    }

    static void foo(String s) {
        System.out.println("It's a string");
    }

    static void foo(Object o) {
        System.out.println("It's an object");
    }
}

This fact answers the "Why?" from above. In the example, the compiler knows that the List is a List<String>, and therefore when the string concatenation is compiled into StringBuilder invocations, it compiles to StringBuilder.append(String) instead of StringBuilder.append(Object). Then at runtime, a java.lang.Integer is passed to the append(String) method, which obviously won't work. The proof is in the disassembled class code. Disassembled code from the example above:

0:   invokestatic    #16; //Method legacyGetList:()Ljava/util/List;
3:   astore_1
4:   getstatic       #20; //Field java/lang/System.out:Ljava/io/PrintStream;
7:   new     #26; //class java/lang/StringBuilder
10:  dup
11:  ldc     #28; //String First element:
13:  invokespecial   #30; //Method java/lang/StringBuilder."<init>":(Ljava/lang/String;)V
16:  aload_1
17:  iconst_0
18:  invokeinterface #33,  2; //InterfaceMethod java/util/List.get:(I)Ljava/lang/Object;
23:  checkcast       #39; //class java/lang/String
26:  invokevirtual   #41; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
29:  invokevirtual   #45; //Method java/lang/StringBuilder.toString:()Ljava/lang/String;
32:  invokevirtual   #49; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
35:  return

Note the "26: invoke StringBuilder.append(String)". That's the bad method call I mentioned. You'll notice the compiler throws a checkcast on the line before, which is where our ClassCastException comes from. If we change the List<String> to List<Object>, we get:

0:   invokestatic    #16; //Method legacyGetList:()Ljava/util/List;
3:   astore_1
4:   getstatic       #20; //Field java/lang/System.out:Ljava/io/PrintStream;
7:   new     #26; //class java/lang/StringBuilder
10:  dup
11:  ldc     #28; //String First element:
13:  invokespecial   #30; //Method java/lang/StringBuilder."<init>":(Ljava/lang/String;)V
16:  aload_1
17:  iconst_0
18:  invokeinterface #33,  2; //InterfaceMethod java/util/List.get:(I)Ljava/lang/Object;
23:  invokevirtual   #39; //Method java/lang/StringBuilder.append:(Ljava/lang/Object;)Ljava/lang/StringBuilder;
26:  invokevirtual   #43; //Method java/lang/StringBuilder.toString:()Ljava/lang/String;
29:  invokevirtual   #47; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
32:  return

The checkcast is gone, presumably because there's no reason to check if something is castable to Object, and it now correctly calls StringBuilder.append(Object).

Cookies

2009-01-31T12:55:00.000-08:00

The web is full of "the best chocolate chip cookie recipe"s, but here is one from a NY Times article that would truly be a grievous loss to civilization if it were to pass out of knowledge. According to the accompanying article, this recipe was created with the input of a number of master bakers from New York City. I believe it. This recipe makes what could be the absolute best homemade chocolate chip cookies I've ever had. The recipe is online, but I'm replicating it here. Note if you don't read the article: refrigerating for 24-36 hours before cooking is a very important step. Also, I used regular semi-sweet chocolate chips, and they turned out fine. Maybe they'd be even better with the fancy chocolate?

Time: 45 minutes (for 1 6-cookie batch), plus at least 24 hours’ chilling

2 cups minus 2 tablespoons

(8 1/2 ounces) cake flour

1 2/3 cups (8 1/2 ounces) bread flour

1 1/4 teaspoons baking soda

1 1/2 teaspoons baking powder

1 1/2 teaspoons coarse salt

2 1/2 sticks (1 1/4 cups) unsalted butter

1 1/4 cups (10 ounces) light brown sugar

1 cup plus 2 tablespoons (8 ounces) granulated sugar

2 large eggs

2 teaspoons natural vanilla extract

1 1/4 pounds bittersweet chocolate disks or fèves, at least 60 percent cacao content (see note)

Sea salt.

1. Sift flours, baking soda, baking powder and salt into a bowl. Set aside.

2. Using a mixer fitted with paddle attachment, cream butter and sugars together until very light, about 5 minutes. Add eggs, one at a time, mixing well after each addition. Stir in the vanilla. Reduce speed to low, add dry ingredients and mix until just combined, 5 to 10 seconds. Drop chocolate pieces in and incorporate them without breaking them. Press plastic wrap against dough and refrigerate for 24 to 36 hours. Dough may be used in batches, and can be refrigerated for up to 72 hours.

3. When ready to bake, preheat oven to 350 degrees. Line a baking sheet with parchment paper or a nonstick baking mat. Set aside.

4. Scoop 6 3 1/2-ounce mounds of dough (the size of generous golf balls) onto baking sheet, making sure to turn horizontally any chocolate pieces that are poking up; it will make for a more attractive cookie. Sprinkle lightly with sea salt and bake until golden brown but still soft, 18 to 20 minutes. Transfer sheet to a wire rack for 10 minutes, then slip cookies onto another rack to cool a bit more. Repeat with remaining dough, or reserve dough, refrigerated, for baking remaining batches the next day. Eat warm, with a big napkin.

Yield: 1 1/2 dozen 5-inch cookies.

Note: Disks are sold at Jacques Torres Chocolate; Valrhona fèves, oval-shaped chocolate pieces, are at Whole Foods.

Good Design in 42 Seconds

2009-01-26T13:48:00.000-08:00

Caution: not for the novice programmer! Ready? Go.

Find a method that's more complex than average. Use "extract method" refactoring to decompose a complex method into smaller pieces: private methods with self-documenting (doesn't-need-comments) names and correspondingly well-defined roles, like createNewThis, storeThat, and processIntoTheOther. Then convert each of the private methods into an interface/implementation pair. Apply this across a project, and you find yourself with pyramids of objects. The very lowest-level business and integration operations are encapsulated by small building blocks at the bottom. Mid-level objects are no larger than the lower, but assemble the lower-level objects into useful sequences for doing work. Higher level objects continue extending this pattern of maintaining small size but assembling functionality from lower-level chunks. Then unit testing any single object is as simple as mocking the few objects it depends on and testing that it invokes its dependencies in the right order and passes them the right parameters under whatever conditions it's in charge of knowing about. Changing behavior is equally easy because simply writing a new implementation of an object that's <100 lines is nothing. The old implementation stays around for reference, including its supporting tests, or it and its tests can be removed. No more changing the code and then having to trace through old tests to see what needs to be fixed there. Polymorphic open/closed.

Ubuntu Video Problems or How I Learned To Stop Worrying and Love Installing ATI Drivers

2009-01-01T13:53:00.000-08:00

I've finally gotten around to fixing my broken Ubuntu desktop. It required a reinstallation of ATI video drivers for my Radeon X1900 card. The Situation: I had installed a fresh copy of 8.04 just after it came out, and I updated via the update manager as updates became available. A couple months ago, I installed some updates which upgraded me to kernel version 2.6.24-19-generic. I don't know what else changed, but after this I couldn't get to the desktop. I removed gdm from the init scripts so I could work at the command line. Every time I tried to run startx, though, it would first go to the grey/black and white check background with the "X" mouse cursor, then to a plain white screen with the normal Gnome mouse cursor. The mouse was responsive, but there was only the white screen. Hitting Alt+Tab would show the outline of what appeared to be a dialog box with the outline of the "task switch" panel in front of it. Hitting enter would make the dialog go away so that this didn't happen anymore and I was just stuck with the plain white screen. Ctrl+Alt+Backspace would make the desktop appear for an instant before dropping me back to the command line. The Resolution: Googling led me in a few different directions, and I tried things like trying to reset my xorg.conf via dpkg-reconfigure xserver-xorg. This generated a basic xorg.conf, but it also failed with this error about a battery: FATAL: Error inserting battery (/lib/modules/2.6.24-19-generic/kernel/drivers/acpi/battery.ko): No such device ??? This is a desktop! Obviously there's no battery! I never figured out what was up with that. Anyway, the new, very small xorg.conf didn't make anything different happen. I also tried updating a couple times along the way with:

apt-get update
apt-get upgrade

Still no dice. Previous experience had taught me that, regardless of which OS you're using, if you have weird problems like this, there's a better-than-even chance that it's got something to do with the video drivers, so I thought I'd try updating those. The Unofficial Wiki for the ATI Linux Driver was very handy in this regard. I found it linked on the ATI driver page when I went there to download the driver. Of course, downloading and installing the driver yourself is considered the Hard Way, but I've done it once before, and it was the automatic way that got me in this predicament. The wiki has instructions for multiple version of multiple distros, including a page for Ubuntu 8.04. I followed the instructions along to the "Install .debs" step before I hit my first problem. The problem was that I had conflicting packages installed because while I was waiting for the driver to download, I had installed Envy with the intent of trying to have it straighten out my drivers for me. I never tried it though, opting instead to handle the installation myself. It turns out, though, that in installing it, several packages had been added that got in the way of the packages that the driver wanted to install. Removing the Envy packages let the driver packages install fine, but at the very end, I got this error message: [fglrx:firegl_init_module] *ERROR* firegl_stub_register failed Further Googling led me to a forum thread that mentioned that trying to install the fglrx module (the ATI drivers) with modprobe fglrx is what causes this error. I tried running that command, and sure enough, I got the same message. The thread also mentioned that modules "radeon" and "drm" seem to get in the way of fglrx, and removing them allows modprobe fglrx to work. lsmod showed that I also had the two modules in question installed, so I crossed my fingers (I had no idea what would happen) and removed them with rmmod radeon and rmmod drm, then ran the entire installation process again, just to be safe. This time it went fine. I started on my way down the checklist again, and when I got to sudo aticonfig --initial -f, I realized that this is something I should have tried in the first place, since this generates an xorg.conf file with all the right ATI stuff in it. Oh well. After finishing the checklist, my desktop is back, and this is the first blog post I've made from Ubuntu. Woohoo! P.S. The title of this post is a reference to one of the greatest movies ever: Dr. Strangelove.

Implementing Optionally Infinite Iterations

2008-12-30T05:17:00.000-08:00

Sometimes you need to write a method that takes an argument indicating how many times a certain thing should be done with the option of doing it an unlimited number of times. A straightforward writing might look like this:

    /** Pass in -1 to work forever. */
    public void doStuff(int iterations) {
        int count = 0;
        while (count < iterations || iterations == -1) {
            foo();
            count++;
        }
    }

If that just bugs you because one of the conditions will always be unnecessary to check--it bugs me--then what about this?

    /** Pass in -1 to work forever. */
    public void doStuff(int iterations) {
        final long maxCount = iterations == -1 ? Long.MAX_VALUE : iterations;
        int count = 0;
        while (count++ < maxCount) {
            foo();
        }
    }

Since an int can never be as much as Long.MAX_VALUE, you've got your infinite iterations, and there's only the one condition to check. All it requires is that you know how big the primitive types are in java and understand how two's complement numbers work.

Object.getClass() and generics

2008-12-12T10:28:00.000-08:00

I ran into trouble with Java's generics (who hasn't?). I couldn't figure out why this wouldn't compile:

class Foo {
   public static void main(String[] args) {
       List<Class<Foo>> list = null;
       list.add(new Foo().getClass());
   }
}

Here's the error (remember, it's a compile-time error): The method add(Class<Foo>) in the type List<Class<Foo>> is not applicable for the arguments (Class<capture#2-of ? extends Foo>) This is what I figured out: First, as far as I know, that "capture#2-of" is just an internal name that the compiler assigns to the generic type and doesn't really have any great meaning, so basically the compiler seems to be complaining that Class<? extends Foo> isn't assignable to Class<Foo>, which is true, of course:

Class<? extends Foo> c1 = null;
Class<Foo> c2 = null;
c2 = c1; // <-- error

Same thing as:

List<? extends Object> l1 = null;
List<Object> l2 = null;
l2 = l1; // <-- error

This is illegal because consider if l1 is actually a List<String>, then:

l2.add(new Date());

would be a legal method call, and you've just added a Date to a List<String>. So we know that Class<? extends Foo> isn't assignable to Class<Foo>, but the question remains: why does the compiler think we're trying to do that? Well, for the next step, what does java.lang.Object.getClass() return? Per the API docs, it's a Class<?>, but that's not the whole story. We can see from our example that it's actually "Class<capture#2-of ? extends Foo>". Now what I couldn't figure out was why the type of Class was "? extends Foo". Shouldn't getClass() called on a Foo object return something of type "Class<Foo>"? But I was forgetting about polymorphism and how it plays with these compile-time parameterized types. In this code:

Foo foo = ...;
Class<...> c = foo.getClass();

the compile-time type of c can't be Class<Foo> because Foo is only the declared type of the variable. At runtime, the foo variable could actually hold an instance of any subclass of Foo, meaning the type returned by getClass() must be at least as wide as Class<? Extends Foo>, which explains this last bit of the puzzle. This is why the result of new Foo().getClass() can't be guaranteed to fit in a variable of type Class<Foo>. For more good stuff on generics, Sun's generics tutorial is very easy to understand.

On a Quest for Social Lending

2008-10-25T10:53:00.000-07:00

(Jump right to the list.)

This "global economic crisis" isn't all bad. If you've got some cash piled up, like Warren Buffet--even if it's not quite the $44 billion he started this year with--you're in a great position to be hunting for opportunities while others are just trying to make ends meet as their credit lines shrink. One such opportunity, in my opinion, is something that recently came to my attention: the relatively new movement called "social lending".

The shortest way to explain it is that people borrow money from other people instead of from banks. Why social lending? Consider how banks make money. First, they buy your money by giving you interest on money you deposit with them in the form of savings and checking accounts, MMA's, CD's, or whatever. Then they turn around and sell your money to someone else at a higher rate: by giving a home loan for 6%, a car loan for 7%, a credit card for 13%, etc. (These are just rough guesses at today's rates.) This is what banks do. It's their purpose for existence and how they stay in business. It's called arbitrage. The question on the lips of social lenders is, "Why should the banks have a monopoly on this?" Social lenders want to get in on this action.

Social lending is also known as P2P (peer-to-peer or person-to-person) lending. As described in the wikipedia article linked above, it can refer to a "marketplace" type of lending, where lenders browse for borrowers they would like to lend to, or a "friend/family" type, where you are lending to or borrowing from a friend or family member and would like a service to help make the loan "official" with appropriate legal paperwork. In this post, I'm referring to the "marketplace" type of lending, in which you, as a lender, actively search for someone to lend money to as an investment. The model I'm primarily interested in is one where borrowers list their loans and lenders "bid" on them with a dollar amount they're willing to give and the interest rate they want for it. In essence, it becomes an interest rate "auction", with lenders bidding down the rate until the "auction period" expires.

Social lending allows anybody with a little bit of cash--as little as $50 in some cases (maybe less?)--to lend that cash out, thereby putting the money to work and getting a higher return than they'd get from a typical bank account. Before you get too excited, social lending sites have taken something of a beating in the past couple of weeks. A quick Google search turned up a WSJ blog and a Washington Post article about how these sites have been affected by recent events: tighter regulation by the SEC and higher-than-normal default rates are putting a dent in their business. Briefly, the sites have been forced to suspend all new business until certain SEC filings are complete--a process that can take months. Unfortunately, this is happening just at the moment when their businesses should be growing by leaps and bounds, as more and more loan-seekers are denied by banks and turn elsewhere to get the funding they need.

A word of caution wouldn't be out of order here: when you loan money like a bank, you're taking on risk like a bank. Don't lend indiscriminately or you'll find yourself in the same position as Wachovia:

"October 23, 2008--Wachovia Corp. reported a $23.9-billion third-quarter loss Wednesday, the largest loss at any bank since the financial crisis began..."

Wachovia, of course, won't be Wachovia for much longer. Lehman Brothers is another good example of how not to lend. I can't find any one, good article, but looking over a news search for "Lehman Brothers" for the past month brings up words like "carnage", "fire sale", "disaster", "subpoenaed", "death", and other unpleasant things. When you lend, choose your borrower wisely, and, as with any financial endeavor, diversify! Split your money across many borrowers to limit your exposure to defaults.

One other thing to check for... To loan money, you have to first have the money somewhere. Make sure that whatever service you go with keeps your money in an FDIC-protected account. If it doesn't, well... I won't tell you what to do with your money, but make sure you know exactly what you're getting into, and weigh the risks appropriately.

If all this doomsaying hasn't scared you off, if you're still interested in the potential investment opportunities offered by social lending, as I am, then you can start by looking through this list of social lending sites I've found. Not all of these are "marketplace" lending sites, but I'm listing them anyway so that it's a complete reference. Further, some of these sites serve particular countries exclusively, so make sure you check that particular detail. I've tried to keep at the top of the list all the sites where a US citizen could go and sign up for an account right now to start lending.

The List

Fynanz - http://www.fynanz.com/ - Social lending that specializes in student loans. This one was going to be further down in the middle of the list (arbitrarily), but I moved it to the top because it seems like a really good prospect to me. Take a look around their site. One thing that makes them stand out is that they recentaly gave referral and lending bonuses. It looks like that's over, but it seems promising for the future. They also have a guarantee system that protects some or all of your investment, which I haven't seen elsewhere. Finally, there's the fact that you're investing in someone's education.

LendingClub - http://www.lendingclub.com/ - A high-profile US social lending site. They very recently reopened lending after completing the aforementioned SEC registration process.

Loanio - https://www.loanio.com/ - Another site that is currently open for business to US residents. I'm not sure if they've had to or will have to register with the SEC. This is something to check on before you decide to go with them, since they might have to shut down lending for weeks or months to get the registration done.

Prosper - http://www.prosper.com/ - Probably the most prominent social lending organization serving the United States. As of this writing, they aren't accepting new loans because of the SEC registration process I mentioned above. They began the process on 15 Oct 2008, I belive, and it's unknown when they'll open back up to lenders.

CommunityLend - http://www.communitylend.com/ - A new player on the field: they're not open for business yet. It seems they intend to have a public beta, and you can be notified when it launches by entering an email address on the "I want to Invest" part of their site. Note: This is a Canadian company, and I can't find any information about who is eligible to be a lender. I've emailed them to find out. Another note: Before I even finished this post, the Chief Technology Officer of CommunityLend wrote me back--on a Saturday!--to say that because of financial service regulations, the service will only be available to Canadians. That's unfortunate. Their website looks promising.

Yadyap - http://yadyap.com/ - Another not-quite-launched site, Yadyap will reportedly be the social lending equivalent of payday loans. I don't know much else about it. You can be notified when they launch by entering your email address on their home page.

Zopa - https://us.zopa.com/ - Included here for completeness, Zopa recently stopped doing business in the US. They apparently still have a booming business in the UK and also offer services in Italy and Japan. Maybe they'll come back to the US. Who knows? The seem to have had a different business model than the typical auction of a "marketplace" social lending company.

GlobeFunder - https://www.globefunder.com/ - I'm not sure if this is the same kind of company as Prosper or LendingClub. They're not operational yet, and the site doesn't give a lot of detail about what they're trying to do. It does mention "individual lenders" on their Lenders page, so maybe it will be what I'm looking for.

Fosik Lending - http://www.fosik.com.au/ - An Australian lending site that does both "friends and family" and "marketplace" lending. The "marketplace" portion is currently in beta testing and will reportedly be made a public beta. I'm fairly certain that only Australian citizens are eligible, but it isn't stated on their website. I've sent an email requesting confirmation.

iGrin - https://www.igrin.com.au/ - Another Australian lending site that plainly states it's open only to Australians. It's too bad. It looks like a well-executed website.

"Friends and family" loans only: Virgin Money - http://www.virginmoneyus.com/ - No marketplace here. Their whole thing is about formalizing and managing loans between friends and family members.

LoanBack - http://www.loanback.com/ - About the same as Virgin Money, from what I can tell.

Finally, something a little different: "marketplace" social lending targeted at enterpreneurs in third-world or poverty-stricken areas. "Enterpreneur" here doesn't necessarily mean someone looking for $10,000 to rent some office space and set up shop. It could just as easily be someone looking for $300 to buy some pigs for a farm. That's not intended to be derogatory. I just want you to know what to expect:

Kiva - http://www.kiva.org/ - Just from poking around the websites to write this post, Kiva seems like the leader in this section. They have a polished site and and seem to have a lot of traffic. Kiva appears to be open to lenders from all over the world, US included. They target poor enterpreneurs all over the world, and the result is a strong tendency toward agricultural borrowing.

MyC4 - http://www.myc4.com/ - This site focuses on Africa. Borrowers seem to tend more toward light industry--textiles, light manufacturing, and the like. It's not clear whether US citizens are eligible, but the front page claims that they have investors from 75 countries. All monetary amounts are in euros.

United Prosperity - http://www.unitedprosperity.org/ - This one isn't operational yet, but seems like it will be similar to Kiva.

How to Process a File Line-By-Line in Linux

2008-10-20T13:15:00.000-07:00

Being a Linux noob, I had to look around for how to process a text file line by line in a shell script. I have no idea if this is the best way, but here's what I figured out:

cat movies | awk 'system("echo " $1)'

where "foo" is the file. That sends the file to awk, and the stuff in the single quotes tells it to invoke the system command "echo" for each line of the file, passing the line to it. You can obviously substitute other commands for "echo", like "wget" to download a bunch of stuff if the file is a list of URLs.

How To: Run Selenium Tests with Hudson on a Headless Linux Server, Part Three--Configuring Hudson

2008-10-15T18:59:00.000-07:00

So you've got Xvfb running, and you've tested it by taking a screenshot or three of xclock. Now to get it working with Selenium tests in your Hudson builds. First, let me say that my work was done with Selenium RC, in which you run a standalone Selenium server which is resposible for launching browsers and receives commands from your test scripts to run in the browsers. I'm not highly familiar with the other varieties of Selenium, so I can't say how similar the setup would be for them.

First, Selenium has to know which browser to start and/or how to start it and which display to use. If you're already using Selenium RC, you'll know that you have to pass a browser command to Selenium to indicate what browser to use. However, if you normally work in Windows, and Selenium can't find Firefox or IE on your Linux box, you might need to do a little more configuration here. A typical browser command to launch Firefox is "*firefox". Selenium has a list of "default locations" where it looks for the Firefox launcher. If it can't find it, you can specify it manually, like *firefox /usr/bin/firefox-bin. This tells Selenium that it's starting a Firefox instance and to use the given path. You must provide the path to firefox-bin and not just to the firefox script. Selenium checks to see if it's been given a script or an executable binary, and it will throw an exception if it finds a script. There's also an option to just pass a path and arguments to Selenium, leaving off the "*firefox" designator, but as the docs say, "If you specify your own custom browser, it's up to you to configure it correctly. At a minimum, you'll need to configure your browser to use the Selenium Server as a proxy, and disable all browser-specific prompting."

It's simple to tell Selenium to use the virtual display. Setting an environment variable named "DISPLAY" in Linux tells any graphical app that starts to start on the specified display, so it's just a matter of getting that variable set properly for the Selenium server process. Remember that it's the server that's responsible for starting the browser, so that's where the DISPLAY variable has to be available. If you're launching the server from a shell, you can just do export DISPLAY=:5.0 before launching the server. Naturally, you'll need to make sure the numbers match up with the display and screen that you configured in Xvfb. (See the first post in this series for details on that.) If you're launching the server with Ant, just add a nested element to the <jar> target that looks like this: <env key="DISPLAY" value=":5.0" />. However, keep in mind that if you add this to your build script, then you're tying all Linux users to that display. That's probably not good. (Been there, done that.)

Finally, make sure that Hudson is orchestrating everything correctly. This might actually be the smallest part of the whole thing since Hudson can invoke shell scripts, Ant targets, Maven goals, and a ton of other things. Make sure that Xvfb is running or that Hudson starts it. Make sure your application server/web container is running or that Hudson starts it. Make sure that any other stuff your application depends on is available, like a database. Make sure Hudson builds your web application and deploys it as appropriate. Make sure the Selenium server is running or that Hudson starts it. Then just have Hudson invoke the target, goal, or whatever that starts your Selenium test suite. Since Selenium tests are written as normal unit tests with JUnit or TestNG or whatever, there's really nothing to that. The only tricky part here is that you need to make sure the tests don't start until your application has fully started up. It's possible that you could start running tests before the URL for your application is even available on your web server.

That should be it! When the Selenium client starts up, it will communicate the start command to the server, which will start a browser in the virtual display. Then the tests will run just like they always do, sending commands to the Selenium server, and in turn to the browser, which makes HTTP requests to your web server, which is running the application that Hudson just built. You can use xwd and xwud, explained in the second post, to take and see screenshots of the browser as your tests are running. A cool idea that I've implemented in our environment is to set up a listener in your test framework (I used an ITestListener in TestNG) that will take a screenshot any time a test fails. This gives you extremely useful feedback to use when checking out the failures.

How To: Run Selenium Tests with Hudson on a Headless Linux Server, Part Two--xwd and xwud

2008-10-14T20:38:00.000-07:00

In Part One, I explained how to start Xvfb to give you a virtual display to run graphical applications on. In this post, we'll verify that it's working properly by starting xclock and taking a screenshot of it. Being able to take a screenshot of a virtual display is a pretty useful skill all by itself. To take and view the screenshot, you'll use two different programs: xwd and xwud. The former takes the screenshot; the latter shows it. These should have been installed with your X server (I think).

First, start xclock on the system where Xvfb is running with xclock -display :5.0 so that it will run on the virtual display. Next, also on the system with Xfvb, take the screenshot with

xwd -root -display :5.0 -out xwdout

The breakdown:

-root: Tells xwd to capture the "root" window. This means it captures the entire screen. You can tell it to only capture a certain window if you want. See the -id and -name parameters in the man page if you want to try that.
-display :5.0: Tells xwd to look at screen 0 of display 5. This is why you have to know that info from when you started Xvfb!
-out xwdout: Gives the name of the output file to write the screenshot to. Xwd uses a custom binary format for its files. They can be read by xwud (next), and I understand that they can be converted to other formats by ImageMagick (free!), though I haven't tried it yet.

After taking the screenshot, you'll need to get the output file to a system that has a real X server running, so you can see the image. Some options for this are ftp or sftp, scp, or good old Sneakernet. Finally, view the screenshot you took: xwud -in xwdout. This shows the image stored in the file named xwdout. It should open a window where you can see a happy little xclock showing the time it was whenever you took the screenshot. Click anywhere in the window to close it.

That's it for this post. In Part 3: how to make Hudson, Selenium, and your build play with all this other stuff you've learned.

How To: Run Selenium Tests with Hudson on a Headless Linux Server, Part One--Xvfb

2008-10-14T19:11:00.000-07:00

I've recently set up Hudson as the continuous integration server for my project at work. I chose Hudson over Cruise Control and Continuum for two reasons: Hudson was highly recommended by a former coworker (thanks Mike!), and, when I was choosing, the Hudson site was much friendlier and easier to navigate. I'm not going to cover setting up Hudson, because it really is as easy as the site makes it sound, and it has tons of built-in tooltips for help. What this series of posts is going to cover is getting Hudson to run your Selenium tests on a headless Linux server. I'll cover the Linux portion in fairly high detail, since I only had some basic Linux experience when I undertook this adventure, and I had to put all this info together myself from my own research. This isn't for total Linux noobs, though. You should at least know how to install packages and navigate around the file system from the command line before reading this.

A note for the uninitiated: Selenium is an open source library for testing web applications at the UI level. It uses JavaScript to interact with web pages, so you can script a series of user actions and ensure that your application functions as expected in the browser. But just as this isn't a series about setting up Hudson, it's also not a series about setting up Selenium.

Let's get down to Part One. Assume you have some Selenium tests in your test suite and you want to get them working from a Hudson-initiated build on your headless server. Headless means there's a good chance you have no X server running, and you can't run Firefox or your browser of choice without an X server, which means you can't run your web application. Instead of starting up a full-fledged X server just to run some UI tests, how about a virtual display instead? That's what a nifty tool named Xvfb gives you. Xvfb starts a very basic, virtual display in memory so that applications requiring graphical capabilities can run on a machine that doesn't routinely run an X server.

Step one is to make sure that Xvfb is installed. Note that this means you have to have an X server installed, too. Yes, Xvfb makes a lightweight, virtual display, but it still uses the X server to do it. Once Xvfb is installed, you should have an executable binary named Xvfb installed somewhere: probably in /usr/bin. I'll assume that's where it is. Next, we'll start it up:

/usr/bin/Xvfb :5 -ac -screen 0 1024x768x8

How this breaks down:

/usr/bin/Xvfb: The executable that actually starts Xvfb. Duh.
:5: Tells Xvfb to use display 5. X servers can have multiple "displays" with multiple "screens" per display. The default display is 0. I chose to use 5, because sometimes a real X server is started on the machine in question (for VNC connections), and I didn't want to find out what would happen if there were a collision. I don't know what the limit is. It doesn't really matter what you use, just as long as you remember what it is. You'll need it later.
-ac: Disables access control to the X server, enabling access by any host. This is Linux-speak for, "You can use this server from any host, local or remote." Per the Xserver man page, "Use with extreme caution. This option exists primarily for running test suites remotely." Since we're doing testing, we're fine.
-screen 0 1024x768x8: Creates screen number 0 on display 5 at resolution 1024x768 and 8-bit color depth. Obviously these numbers can be whatever you want.

Now Xvfb should be running and ready to invisibly display stuff for you. The values you set for "display" and "screen number" are important, and are combined in the form ":." to specify to Linux which display/screen combination to use for displaying things. In the example above, you would use ":5.0" to tell Linux to use the single screen that is configured. I should also add that it doesn't seem to matter which user starts Xvfb, so you can create a startup script in /etc/init.d and let it run as root if you like. You can test your Xvfb by starting a graphical application and taking a screenshot. That's what I'll cover in Part 2--xwd and xwud.

Unlocked

2008-10-10T06:25:00.000-07:00

Well, this blog got automatically locked as a potential "spam blog", or I would've posted a couple of things over the past few days. I suspect the robots noticed the total disregard for the English language in my last post and locked me down for it. I'll post something interesting later today maybe.

An Unexpected Chapter

2008-10-02T01:12:00.000-07:00

I certainly didn't expect to be awake now, but a coworker is out of the country and working from a very different time zone, and he IM'd me with a question at about midnight, my time. The question was related to a problem we'd been seeing related to Apache Camel 1.4. There was a NullPointerException coming from inside Camel somewhere. This is the exception and stacktrace:

Execution of JMS message listener failed                                                                                 WARN  DefaultMessageListenerContainer(634) 01:25:03,534 DefaultMessageListenerContainer-10
org.apache.camel.RuntimeCamelException: java.lang.NullPointerException
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:71)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:531)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:466)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:316)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:887)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:822)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
at org.apache.camel.component.jms.JmsMessage.createMessageId(JmsMessage.java:203)
at org.apache.camel.impl.MessageSupport.getMessageId(MessageSupport.java:127)
at org.apache.camel.impl.MessageSupport.copyFrom(MessageSupport.java:95)
at org.apache.camel.impl.DefaultExchange.safeCopy(DefaultExchange.java:98)
at org.apache.camel.impl.DefaultExchange.copyFrom(DefaultExchange.java:81)
at org.apache.camel.impl.DefaultEndpoint.createExchange(DefaultEndpoint.java:145)
at org.apache.camel.component.file.FileProducer.process(FileProducer.java:54)
at org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:43)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:75)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:57)
at org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:155)
at org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:91)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:85)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:57)
at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:39)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:41)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:66)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:68)
... 8 more

He IM'd me to say he'd discovered that trying to use two file endpoints in the same camel route caused this exception. After some testing, I narrowed his theory to "using a file endpoint anywhere but the end of a route". I should say that this is in a route that processes JMS messages. Here's the problem: Camel pulls a message from a JMS desintation and sends it along the route as an org.apache.camel.component.jms.JmsMessage packaged inside an org.apache.camel.component.jms.JmsExchange. When you get to a file endpoint, it does this:

1 public void process(Exchange exchange) throws Exception {
2     FileExchange fileExchange =
          endpoint.createExchange(exchange);
3     process(fileExchange);
4     ExchangeHelper.copyResults(exchange, fileExchange);
5 }

That's in org.apache.camel.component.file.FileProducer, for those following along. This is what creates the file from the incoming message/exchange. Line 2 there creates a FileExchange containing FileMessages based on the incoming JmsExchange, which contains JmsMessages. Line 3 is what writes the file using the contents of the newly created exchange. Then, for whatever reason, Camel copies the state of the FileExchange back onto the incoming exchange (yes, that method is copyResults(destination, source)), and that's the exchange that gets passed on down the route. In ExchangeHelper.copyResults, the message from the source exchange (the out message if it exists, the in message if out doesn't exist) is copied to the destination's out message. The destination doesn't have an out yet, so one is created, and then the state of the source message is copied to it. The problem is that JmsMessage has a property of type javax.jms.Message where it stores the original JMS message that it pulled from the JMS broker. FileMessage obviously does not have this. Basically what the above code does is copy from JmsMessage -> new FileMessage -> new JmsMessage. Plainly, then, any state in the original JmsMessage that doesn't fit in FileMessage will be lost in the new JmsMessage. One of those is the javax.jms.Message property. Unfortunately, this property is an integral piece of Camel's JmsMessage. One place it's used is in generating a message id in the createMessageId method, which you'll notice is the top line in the stack trace for the NullPointerException above, and the message id is used all over the place. What this boils down to is that you may not use a file endpoint in a route where the message comes from a JMS server except as the very last endpoint.* However, you can fudge this a little using Camel's implementation of the multicast pattern. This will break:

from("someJmsEndpoint")
    .to("file://foo").to("file://bar");

But this works:

from("someJmsEndpoint")
    .multicast().to("file://foo").to("file://bar");

Each file endpoint still has to be at the very end, but multicast() effectively lets you have many "ends" to a single route. *This may be a problem with mixing other message and exchange types as well, but JMS/File is the only one I've seen break myself.

The First Chapter

2008-10-01T19:10:00.000-07:00

Here's an interesting problem I came across the other day related to transaction demarcation.

I'm writing some message-driven code using a JMS broker--specifically, ActiveMQ: http://activemq.apache.org/. A lot of the processing is actually running in the broker JVM itself using Apache Camel to move messages around: http://activemq.apache.org/camel. We recently had trouble with some of the processing code when we first deployed the message broker to a staging environment and tried to push several thousand messages through it. It ran through a bunch of the messages just fine, then it stopped--no errors in the log, no crash of the broker, no CPU use... nothing. It just sat there grinning stupidly at me. Here's how the message processing works...

New message comes in. Get the important details out of it and invoke a processMessage method that does everything needed to put the message into our database. The database is Oracle in staging and production but Apache Derby in development. The processMessage method is wrapped with a typical transaction using Spring's declarative transaction management. Inside that method, another method is invoked that’s configured with PROPAGATION_REQUIRES_NEW. There are a few factors working together that make this necessary, but they’re not relevant here.

It goes something like this:

Begin outer transaction/open Hibernate session 1
Do some transformations and creation of object A to represent message
Get related object B from database or an external source
If object B came from outside, save it with session.save(B) (session 1)
Associate object A to object B
Begin inner transaction/open session 2
Save object A with session.save(A) (session 2)
Commit inner transaction/close session 2
Attach object A to session 1 for continued processing
Maybe make other changes to object A
Commit outer transaction/close session 1

Here's how the problem manifested...

When the broker froze up, the logs stopped right after a SOAP call was made for object B and we tried to save it. Significantly, it was the first time since the broker was started that we’d had to make a SOAP call and save the result to the database. The last thing the logs record is that B was retrieved and saved successfully, but it never made it into our database. We ruled out the possibility that the problem was in the data being retrieved. It was definitely a problem with the code running in the broker.

That’s it. Can you figure out what’s wrong? Read on for the answer.

Hint 1: Object B has unique constraints on some of its fields in the database.

Hint 2: I’ve left out some details about how objects A and B are mapped. I didn’t have the information on hand when I was analyzing the problem and was forced to make some assumptions that allowed me to reach a solution. I still don’t know exactly how they’re mapped, because I already fixed the problem without looking at the mappings. Assume that the relationship from A to B is mapped at least as cascade=”save-update”, and therefore object B will be saved when object A is saved in step 7 above.

Hint 3: It’s all in the transactions. Look there.

What’s going on: I made one other assumption to explain things related to the session.save(B) call in step 4 above. That call will cause an insert statement to be issued for B at some point. My assumption was that, for some reason, the session is being flushed and this insert is happening before the inner transaction is begun in step 6. (I can’t give you any more information on this right now, as I never verified this assumption, either.)

Given these two assumptions, here’s what’s happening at the database level:

Open connection 1, autocommit off
Insert B on connection 1
Open connection 2, autocommit off
Insert A on connection 2
Insert B on connection 2
Commit on connection 2
Commit on connection 1

Do you see the problem now? If not, you probably need one more crucial piece of information: earlier uncommitted statements against an Oracle database will block later, conflicting statements from being committed until the earlier one is committed or rolled back. More specifically, if you try to insert object B on two different connections simultaneously, the second one can’t know if it should succeed or fail until the first one is committed--remember B has unique fields! If you look back at my sequence of database events, you’ll see that I was trying to commit the second connection and then the first connection.

Tie all this together, and you have a deadlock between a single thread and the database. Connection/transaction 2 waits for connection/transaction 1 to commit or rollback, and connection/transaction 1 is unable to do either because execution is stuck waiting for connection/transaction 2 to finish doing what it’s going to do.

Nice, huh?

Solution: Make sure object B only gets associated to one session, the one tied to the inner transaction, so that it’s inserted only one time--along with object A.

Final note: This problem never cropped up in development because Derby apparently doesn’t care about database locks. It seems to blow right past many of them without batting an eye. I discovered this while experimenting with ActiveMQ using JDBC persistence in a Master/Slave failover configuration.

Prologue

2008-09-30T19:03:00.000-07:00

Okay, first the name... I spent many long, laborious hours pondering several insightful and meaningful names for this blog. They were all taken. Most of them were taken by blogs that had been empty for years. I landed here: two random words which combine to produce a vaguely amusing, somewhat unsettling, and highly memorable mental image which is not the least bit relevant to the subject of this blog. The subject of this blog will largely be my work--enterprise software development, currently in Java--and the interesting problems I encounter along the way. To a lesser extent, there will likely be other random remarks of interest to people who know me personally, people who like computers, or people who are just plain odd.

Thank you; drive through.