Rocket Surgery

I have an open-source project called Project PEA that is an initial attempt to gather metrics on how directly and intentionally classes are tested.

One of the main weaknesses of existing code coverage tools is that all code that executes during a test run is considered covered, and covered equally. If a method under test executes and calls another method for which no test was specifically written, the other method looks like it was tested, because it has coverage. The situation is exacerbated the more that one method calls another. Sometimes method calls can be chained many, many levels from one test, and a method that never had a test written for it has 100% coverage.

Project PEA watches your JUnit tests run and sees how many stack frames are between each class and the unit test being run. So for example, if test testFoo() calls foo(), and foo() calls bar(), and bar() calls baz(), it will track that testFoo() tests foo() at a “test distance” of 1, bar() at a test distance of 2, and baz() at a test distance of 3. PEA collects how many times each method is called at a given test distance, and weights the “testedness” of each method based on a formula that improves the score for methods that are tested at a shorter distance.

Here is a powerpoint discussing the aims of project PEA that I gave at the November meeting of the Southeast Virginia Java Users Group.

This entry was posted on Thursday, January 31st, 2008 at 7:49 pm and is filed under Software development.You can follow any responses to this entry through the RSS 2.0 feed.You can leave a response, or trackback from your own site.