Coverage testing with Emma

Among the tools I downloaded and set up while working on my stack testing (links to the files I talk about in this post can be found there) is one called EMMA. It is a coverage tool which goes together with JUnit testing and reports to the user how much of the overall code was actually seen by the test cases. Here is a part of the sample output from EMMA:



It shows, for example, that only 50% of the methods in ClearStack were actually tested, 39% of the code blocks, and 50% of the lines in the class. EMMA doesn't give the user any information about how useful the actual testing was but, as a minimum, it does show that there is much more testing to be done.

Working toward 100%

I started out with the Stack.java testing and simply added tests for the toString method and a test of the top method on an empty stack. This brought my EMMA report up to 100% complete for the Stack class simply by adding two rather trivial tests. This just means I covered the baseline. I know that at least one test was done on all portions of the code.

From there I started to improve on the TestClearStack class. One of the first problems I ran into was not with the ClearStack class but in the testing class. It did not test any methods that could potentially throw exceptions, and didn't have a throws EmptyStackException in its declaration so Eclipse immediately started to complain when I started adding additional test code. The addition of a few tests for isEmpty and getTop caused Eclipse to become upset that it could not resolve ClearStack and EmptyStackExceptions to a type. This error seemed very odd to me, as JUnit and EMMA were still running just fine and my coverage was improving. I was pleasantly surprised to find that EMMA reminded me to check for both true and false conditions on the isEmpty method. However, for the other errors, I decided to run ant -f verify.build.xml and discovered that a slew of new CheckStyle errors had crept into parts of my code which I didn't even think I'd touched. Slightly baffled, I corrected those errors, made Eclipse and CheckStyle happy, and was eventually able to verify the entire project. Here is a link to the final distribution.

Conclusion

Overall the whole experience with EMMA was minimal. I didn't find any new coding errors in the Stack and ClearStack classes while striving toward 100% coverage. The toString method seems odd to me though. I would expect it would want to print out as "[1, 2, 3]" like similar collections, as opposed to "[Stack [1, 2, 3]]", but the documentation did not really mention the expected output so I left it as is. Most of the learning was in dealing with a number of issues with CheckStyle and Eclipse, plus a little more experience with setting up my project workspace correctly. For example, my first distribution build ended up being called "stack-aricwest-6.0.23-6.0.25" simply because of the way I imported my previous code. Even though I didn't find EMMA particularly useful at this point, I definitely see the potential for finding untested code in larger projects. One must remember that it is only there (from my limited experience) to identify a bare minimum of testing and doesn't indicate that a project has been thoroughly tested simply because it reached 100% coverage.