Coverage testing with Emma

Among the tools I downloaded and set up while working on my stack testing (links to the files I talk about in this post can be found there) is one called EMMA. It is a coverage tool which goes together with JUnit testing and reports to the user how much of the overall code was actually seen by the test cases. Here is a part of the sample output from EMMA:



It shows, for example, that only 50% of the methods in ClearStack were actually tested, 39% of the code blocks, and 50% of the lines in the class. EMMA doesn't give the user any information about how useful the actual testing was but, as a minimum, it does show that there is much more testing to be done.

Working toward 100%

I started out with the Stack.java testing and simply added tests for the toString method and a test of the top method on an empty stack. This brought my EMMA report up to 100% complete for the Stack class simply by adding two rather trivial tests. This just means I covered the baseline. I know that at least one test was done on all portions of the code.

From there I started to improve on the TestClearStack class. One of the first problems I ran into was not with the ClearStack class but in the testing class. It did not test any methods that could potentially throw exceptions, and didn't have a throws EmptyStackException in its declaration so Eclipse immediately started to complain when I started adding additional test code. The addition of a few tests for isEmpty and getTop caused Eclipse to become upset that it could not resolve ClearStack and EmptyStackExceptions to a type. This error seemed very odd to me, as JUnit and EMMA were still running just fine and my coverage was improving. I was pleasantly surprised to find that EMMA reminded me to check for both true and false conditions on the isEmpty method. However, for the other errors, I decided to run ant -f verify.build.xml and discovered that a slew of new CheckStyle errors had crept into parts of my code which I didn't even think I'd touched. Slightly baffled, I corrected those errors, made Eclipse and CheckStyle happy, and was eventually able to verify the entire project. Here is a link to the final distribution.

Conclusion

Overall the whole experience with EMMA was minimal. I didn't find any new coding errors in the Stack and ClearStack classes while striving toward 100% coverage. The toString method seems odd to me though. I would expect it would want to print out as "[1, 2, 3]" like similar collections, as opposed to "[Stack [1, 2, 3]]", but the documentation did not really mention the expected output so I left it as is. Most of the learning was in dealing with a number of issues with CheckStyle and Eclipse, plus a little more experience with setting up my project workspace correctly. For example, my first distribution build ended up being called "stack-aricwest-6.0.23-6.0.25" simply because of the way I imported my previous code. Even though I didn't find EMMA particularly useful at this point, I definitely see the potential for finding untested code in larger projects. One must remember that it is only there (from my limited experience) to identify a bare minimum of testing and doesn't indicate that a project has been thoroughly tested simply because it reached 100% coverage.

More lessons from CodeRuler and QA tools

Having had my first taste of Java's available quality assurance tools, it was time to compare it to some of my own code. Having learned quite a lot about improving my coding style and efficiency during the CodeRuler project, it was time to see what automated tools could find. I knew about many of the poor areas in the code due to ignorance or lack of time and proper planning, though the project itself ran quite well. Along with what I learned while working with my project partner, Arthur Shum, I also got the input of two classmates, Robin Raqueno and Ronn Reeves. So I didn't expect too many surprises by running PMD, Checkstyle, and FindBugs on the MyRuler class I helped write.

FindBugs actually reported no errors which at first made me ask myself, "What does FindBugs do again?" It just seems nothing quite fit what it was looking for. That or it wasn't looking in the right place, though the report seemed to be using the correct directory.

Checkstlye returned a huge number of errors, whoever most of this was due to provided IBM code, and only thirteen in my team's contribution. Ten were simply code lines over 100 characters, two were very small white space violations, and the last was a import statement using .* that was included in the original code.

The final check, PMD, was by far the most informative of the three. It found 23 violations, which were significantly more interesting than those found by Checkstyle. Though I was aware of many of these violations either on my own, or from other human reviews, PMD was very thorough and informative. It's suggestions for variable usage were most common. Other mistakes included improper use of comparison operators and not using the more abstract collection interfaces instead of TreeMap and ArrayList.

In the end, these tools are very good at spotting large numbers of errors where human eyes might glaze over, but they need to work as a supplement to peer reviews. Many sections of overall poor code design are quickly noticed by reviewers that automated quality assurance did not bother to note. It was fine by machine readable standards, but most humans would consider many sections of the MyRuler code to be of poor readability. We should use all the tools available to us to write and test the best possible code, both computerized tools and people at every step of the development process from programmer to user.

It's not a bug; it's an undocumented feature.

One of the reasons that Java has become such a popular and widespread programming language since its introduction doesn't actually come from the features of the language itself. Instead, some of its growing usage can be attributed to the vast number of third party tools that are available to make the Java program development process smoother, more efficient, and with less undocumented features. I've recently started familiarizing myself with some of these Java-oriented tools such as the Eclipse platform, Ant, PMD, FindBugs, Checkstyle, and JUnit. Even with just a small amount of experience, I've already starting to think about how I have been able to get along without such tools and guidelines.

How do they stack up?

After downloading and unpacking Emma, JUnit, Checkstyle, PMD, FindBugs, and SCLC the next step is inserting them into the environmental variables listing of my computer so that Ant will know where to look while it is building your project. With that finished I downloaded a test project supplied by my software engineering professor, Dr. Johnson. It's a simple stack class and test cases which are purposefully broken in various ways. Using these new tools, I plan to get the stack class into both a working as well as efficient and consistent system. By running Ant for each of the build configuration files I find that each except "ant -f verify.build.xml" are successful. At first I think that means each one found no errors in the stack project. However, successful means that PMD, Checkstyle, etc. didn't have any problems running - all my paths were correct - however there were plenty errors to be dealt with and generally nice HTML documents to glean over.

Results and repairs

• Checkstyle found two violations of our style standards: a misplaced curly brace and a missing javadoc annotation in stack.java. Both easily fixed.
• FindBugs reported several places that could be changed to be more efficient by changing Integer constructors to Integer.valueOf methods.
• JUnit found the first true errors in the logic, at this point just in the test cases. Code was corrected to reflect a proper pop order of 3, 2, 1. At this point I also caught a typo in the test cases which had no effect on performance, but wasn't caught by other tools. However upon rerunning the JUnit test I still was finding errors and it took some time to discover that the test cases weren't just written wrong, the stack class itself was incorrectly coded in the top function, which was actually looking at the bottom of the stack.
• PMD recommended setting some private variables to final and simplifying a return to get rid of an unnecessary temporary variable. It also reminded me to abstract away from unneeded details such as using a List interface even though an ArrayList is being implemented. The final tricky part was an empty catch block. According to my Elements of Java Style book I could simply use a e.printStackTrace, however Dr. Johnson recommended eliminating the try/catch entirely. I implemented this change as well as cleaned up the unnecessary import of the Assert.fail from JUnit.

At this point I gave "ant -f verify.build.xml" a try and behold: my stack distribution.

Overall it was a good experience with some snags but not too much frustration. The various tasks all seemed to have worked in the end. This is of course only a small part of Ant and the associated build tools. I only had to run them, not really understand them, let alone write them, yet. However I got a good appreciation of what they are capable. I expect it to be a steep learning curve, but see the potential for time saved in the long run. I also expect to be able to apply some of the concepts to other non-Java projects when possible.

Review of another CodeRuler implementation

In an effort to further familiarize myself with the coding practices that I will be using throughout my Software Engineering course (and beyond!), I have been tasked with reviewing one of the other CodeRuler projects submitted to the class. I will be reviewing the MyRuler.java class of Tyler Wolff and Daniel Tian. I downloaded the zip file containing the class and its Javadocs from this link on September 10, but had some difficulties with the zip file and was unable to properly unzip the Javadocs due to a corrupted file. I wasn't able to determine the cause of this error, but was able to extract the MyRuler.java file without troubles.

During our Wednesday class period I was very excited to see this implementation in action as Tyler and Daniel's blogs showed it to be very successful in their tests. Unfortunately it seemed to have encountered a problem during the demonstration and failed to successfully perform any moves. However testing on my own computer was fine and it was perfectly capable of performing up to the standards claimed. In fact it is the only of the implementations I saw that could compete and quite possibly surpass the implementation of Arthur Shum and myself.

Despite not being able to convince my computer to open the javadocs, they were luckily available right in the code and appeared to be thorough. The structure and methods used in the class made sense. Each method was dedicated to a specific task and each well explained. The orderSubjects method is simple and is able to delegate the complex tasks of the implementation to other methods but outlines the workings of the class. The inline comments within various methods help with understanding the logic of the program quickly, though some were redundant. Since many of the variables names consist of three or four words, the code for certain tasks get a bit unwieldy and a number of lines cross well over 100 characters in length. However the logic is quite solid and it's performance outside of the class demonstration is superb.

For the most part the formatting and style of the code is quite close to what is put forth primarily by The Elements of Java Style by Allan Vermeulen, et al. The following chart demonstrates the lapses in proper coding style I observed.

File	Lines	Violation	Comments
MyRuler.java	1		Use of * in import.
MyRuler.java	16-28		Possibly excessive HTML code in javadoc comments.
MyRuler.java	7, 60, 183, *	EJS-49	Omit subject in describing methods.
MyRuler.java	54		Explain empty method.
MyRuler.java	82, 123, 130, *	EJS-7	Closing bracket not on its own line.
MyRuler.java	87		Possibly confusing end-line comment. End of if? or end of else?
MyRuler.java	79, 84, 90	EJS-7	No space between for and (.
MyRuler.java	109		No period ending description.
MyRuler.java	120, 121, 127, *		Four space indent instead of two.
MyRuler.java	120, 127, 134, *	EJS-61	Unnecessary end-line comments.
MyRuler.java	210		May want to use parentheses to seperate logic operations.
MyRuler.java	211, 212, 215, *		Way over 100 characters in these lines.
MyRuler.java	239, 365, 427, *	EJS-7	Beginning bracket for else statement should be on same line as else.

The errors found do not have a huge impact on the readability and reliability of the program. Most can be quickly repaired now that all of the members of the class are more aware of the strict style we should to be adhering to. There is ample commenting of the code, and perhaps too much in several areas where they do not really offer any additional information. Overall though it is an impressive strategy and implemented relatively cleanly and efficiently.

Lessons learned from CodeRuler

Intro to CodeRuler

As a bit of a warm-up exercise to get my software engineering class acquainted and reacquainted with Java and Eclipse we were recently assigned to work on a project using the CodeRuler plug-in from IBM. The plug-in contains a fairly complete framework of Java classes that implement a simple game of knights, peasants, castles, and their rulers. World domination is, of course, the goal. Everything is in place except for the implementation of a strategy. This is where my classmates and I come in. So my partner, Arthur Shum, and I set out to create an AI that would give the provided computer strategies a solid challenge and then later this week we set out to conquer the avatars of our classmates as well.

Teamwork

Arthur and myself started off with a slightly disorganized approach to the project. Initially we just wanted to play around and get familiar with the tools available in the CodeRuler interfaces. From there we individually progressed to various simple strategies and incrementally improved certain ideas and tossed out those that didn't work. Eventually I dedicated myself to programming the behavior of the peasants while he worked on strategy for the knights. I'm proud of my little peasants, but I must admit, his knights are quite impressive and I felt as though I may not have contributed as much to the team. Nevertheless, we worked well together despite not meeting up in person. Communication was done online; we would simply email changes to the code back and forth as we discussed strategies, problems, and solutions on gtalk. Eventually, though we both have numerous ideas that we would like to implement, we got to a point where our strategy was quite successful against the provided computers. We feel it will also be competitive against other teams in the class.

Results

Here are some of the results versus the provided AIs.

	aricwest-atshum	Migrate Ruler
Test 1	829	0
Test 2	822	0
Test 3	827	0
		Gang Up Ruler
Test 1	804	48
Test 2	792	100
Test 3	779	112
		Split Up Ruler
Test 1	805	64
Test 2	827	14
Test 3	801	44

You can find the product of our programming efforts, as well as associated javadocs, here.

Strategy

The basic strategy can be broken up into three parts - one for each unit.

First we have the knights. Arthur came up with a two-team knight approach that sends a small team of knights to capture castles, while the large team goes for enemy knights. To help protect our own castles, the large team prioritizes knights that are closer to our castles. Once the large team wipes out all the knights then it will work on capturing castles, then capturing peasants. The small team instead captures castles, then goes after peasants, then knights. Of course, if either team comes across something else in their path to their main objective they won't hesitate to capture that too. There are some slight variations, but this is the essence of the knight strategy.

Next we have the peasant strategy which I implemented. At first it looks like Random Ruler, and it certainly started out that way. However our peasants are a bit more aware of their surroundings. A peasants basic strategy is to examine the squares around him and randomly pick one that has not been claimed by our team, and is otherwise unoccupied and available. This on its own gives the peasants a quick and rapid start. At some point a peasant will likely become surrounded by land it or another peasant has already captured for our team. At this point our peasant will start moving toward an enemy peasant, since that area will obviously have land belonging to the enemy team. It is also programmed in such a way that our peasants will divide up and chase after different enemy peasants, not cluster after one. Also, if no peasants are available to chase, it will move toward enemy castles. These chase behaviors only last as long as there's no new land to capture. At any point there is new land to capture, the peasants continue with the prime objective - adjacent land. Unfortunately the peasants lack the code to help them evade enemy knights.

The third part of the strategy is simply whether castles should produce knights or peasants. Arthur came up with a simple and effective strategy. Our castles produce knights until we have 50% more than there are enemy knights, then switches to peasants for as long as the 50% more rule holds. A basic overpower-the-enemy strategy, which works thanks to the efficient gathering of land by our peasants, so more are not needed immediately.

Lessons Learned

Over all it was a decent warm-up to Java and Eclipse, both of which I do not have any recent or extensive experience with. However, I think that I didn't take advantage of many features of both that might have made the process easier. Part of it is just not realizing what is available so I work with things I know and understand without branching out enough. I definitely enjoyed using some of the features that had been discussed in our recent software engineering class including the for each and some of basic work with Collections. Working with a partner worked out well and favorably despite never actually meeting in person. In the future I hope to avoid some of the difficulties of emailing our code changes back and forth to merge our additions and changes. An SVN type system would definitely be preferable, but this was a small enough project as to not be a large issue. Our end product is competitive, but I definitely feel it could use some refining in several areas. I would like to add some knight evasion code to the peasant AI. Also there have been time out issues when large numbers of units are in play. Some code could be added to tone down movement when the time out limit is being approached. I'm sure there are also some code that could be more efficient which would help keep the processing time down. Overall though I am pleased with the results and process of this experience.