An incomplete Emma example

I mentioned in my previous entry that Emma is a good tool to see the coverage of JUnit tests. However, it is only a guideline, not an absolute measure of good testing. Simply being able to claim 100% coverage does not mean that all possible states in the program have been tested or that the tests were formulated correctly.

As a simple, if rather contrived example, this distribution contains an artificial error in the pop method of the stack class. By simply inserting a modulo four into part of the code it operates normally for pops when the stack is of small size, and Emma continues to report 100% coverage. While pushing three objects on the stack and popping it once is fine, pushing five objects on to the stack and popping once, which could be considered an equivalent class of testing, it might fail, depending on the pushed object. It will fail if the showNewError test in the TestStack class is not ignored, but the other previous test suite did not catch this error.

Though it is a rather contrived bug, it shows how important thorough testing is and that a programmer cannot simply rely on coverage tools to indicate good testing has been done. Such tools can tell you that tests are missing, but does not tell you if you're really done testing. That requires 110% effort and a report of 500% coverage or more!