Rocket Surgery

Are unit tests and unit testing on the decline?  Here is an interesting blog post that discusses this very question.

My $0.02:  Read the rest of this entry »

At my workplace we’ve started writing JUnit tests to monitor our systems and the systems of the other departements. One unit test pings all the servers we talk to or administer; one does an HTTP get from all the web servers we run, as well as the servers whose web services we call; another test does a quick query against all the databases we hit. We’ve set these JUnit tests up in Hudson to run every ten minutes; one failed test makes the project go unstable, and we get notifications by email and get on top of problems before the other departments notice it. They have their own monitoring software, which is pretty good, but we aren’t given access to it, and this way we get independent confirmation of what they are checking on.

We got this idea from something similar we did at a client of mine in a previous gig. To do cascading delete referential integrity, the DBA’s needed^* to use a delete trigger. Well, any time they made a change to the underlying table, they had to take the trigger off, make the change, then put the trigger back. Guess how often they remembered to put the trigger back. Anyway, we got so tired of them forgetting this and having to fix stuff all the time, that we wrote a unit test to make sure that deletes always cascaded like they were supposed to. Anytime they didn’t, we knew they dropped the trigger, and we got on the phone and fussed at them.

So, on the agenda for our suite are other neat things, such as connecting to our Dev, QA, and Prod databases, grabbing the metadata, and looking for subtle differences like missing indexes from one environment or another. We plan to make sure that the JBoss configuration files (such as connection pool definitions, system properties, JVM memory settings, etc) match what we have saved a copy of so we can tell if someone changes the server’s configuration without telling us about it. We’ll check Java versions, OS versions, etc. Anything that is external to our apps that can affect the success of those apps, we’ll make tests for it. We’ll have executable documentation of our assumptions.

After all, the real purpose of a JUnit test is to assert that something is true; and if it’s not, you’ve got a problem. Nothing says that what you assert to be true has to be the behavior of code — it could be almost anything.

^*I seriously doubt that they had to do it that way, but that was the way they insisted on doing it.

I recently set up StatSVN at my workplace. It’s a great tool for getting statistics from your Subversion repository - which in case you didn’t know, has a great deal of rich data regarding your team’s activity that can be extracted by analyzing source code activity. You can get very nice graphs of activity over time, check-in activity by author, churn (the amount of repeated check-in of the same files), proportion of adds versus updates, even things like which days of the week or hours of the day get the most repository activity.
Read the rest of this entry »

I have an open-source project called Project PEA that is an initial attempt to gather metrics on how directly and intentionally classes are tested.

One of the main weaknesses of existing code coverage tools is that all code that executes during a test run is considered covered, and covered equally. If a method under test executes and calls another method for which no test was specifically written, the other method looks like it was tested, because it has coverage. The situation is exacerbated the more that one method calls another. Sometimes method calls can be chained many, many levels from one test, and a method that never had a test written for it has 100% coverage.

Project PEA watches your JUnit tests run and sees how many stack frames are between each class and the unit test being run. So for example, if test testFoo() calls foo(), and foo() calls bar(), and bar() calls baz(), it will track that testFoo() tests foo() at a “test distance” of 1, bar() at a test distance of 2, and baz() at a test distance of 3. PEA collects how many times each method is called at a given test distance, and weights the “testedness” of each method based on a formula that improves the score for methods that are tested at a shorter distance.

Here is a powerpoint discussing the aims of project PEA that I gave at the November meeting of the Southeast Virginia Java Users Group.

While preparing a training class this past month on Effective Exception Handling for a client, I came up with a list of “Ten Commandments of Exception Handling.” Some of them were the basic, standard items (”Thou shalt not catch or throw java.lang.Throwable”, “Thou shalt not throw raw java.lang.RuntimeExceptions–thou shalt throw a subclass instead”, “Thou shalt not catch an Exception unless thou art going to truly handle it, or are going to wrap it in another Exception”). Other items on the list were specifically tailored for the client. I liked one of the “commandments” better than the others, however:

“XIII: Thou shalt not catch any Exceptions in JUnit tests, unless testing that your code under test throws the Exceptions the API says it does.”

If you read this and say, “well, duh, Matt!” then you and I are thinking alike and I am preaching to the choir. If this statement has even a whiff of controversy to it, however, read on.

Read the rest of this entry »

In a previous blog entry I mentioned that I like emma for measuring code coverage in my JUnit test suites. In this entry I will discuss several items that make some JUnit test suites stronger than others. In this entry I am discussing JUnit 3.8.1.

How do you measure the quality of your JUnit tests and put a number
on it? This is a very good question with no quick and easy answers,
I’m afraid. I can tell you two answers that are NOT measures of test quality:

• “We have 100% success rate in our test suites every night.”
• “We have 100% code coverage”

Read the rest of this entry »