Rocket Surgery

At my workplace we’ve started writing JUnit tests to monitor our systems and the systems of the other departements. One unit test pings all the servers we talk to or administer; one does an HTTP get from all the web servers we run, as well as the servers whose web services we call; another test does a quick query against all the databases we hit. We’ve set these JUnit tests up in Hudson to run every ten minutes; one failed test makes the project go unstable, and we get notifications by email and get on top of problems before the other departments notice it. They have their own monitoring software, which is pretty good, but we aren’t given access to it, and this way we get independent confirmation of what they are checking on.

We got this idea from something similar we did at a client of mine in a previous gig. To do cascading delete referential integrity, the DBA’s needed^* to use a delete trigger. Well, any time they made a change to the underlying table, they had to take the trigger off, make the change, then put the trigger back. Guess how often they remembered to put the trigger back. Anyway, we got so tired of them forgetting this and having to fix stuff all the time, that we wrote a unit test to make sure that deletes always cascaded like they were supposed to. Anytime they didn’t, we knew they dropped the trigger, and we got on the phone and fussed at them.

So, on the agenda for our suite are other neat things, such as connecting to our Dev, QA, and Prod databases, grabbing the metadata, and looking for subtle differences like missing indexes from one environment or another. We plan to make sure that the JBoss configuration files (such as connection pool definitions, system properties, JVM memory settings, etc) match what we have saved a copy of so we can tell if someone changes the server’s configuration without telling us about it. We’ll check Java versions, OS versions, etc. Anything that is external to our apps that can affect the success of those apps, we’ll make tests for it. We’ll have executable documentation of our assumptions.

After all, the real purpose of a JUnit test is to assert that something is true; and if it’s not, you’ve got a problem. Nothing says that what you assert to be true has to be the behavior of code — it could be almost anything.

^*I seriously doubt that they had to do it that way, but that was the way they insisted on doing it.

This entry was posted on Monday, April 21st, 2008 at 6:24 pm and is filed under Software development.You can follow any responses to this entry through the RSS 2.0 feed.You can leave a response, or trackback from your own site.