DueDates 2.0: Herald of Aric 2.0

My final project in Software Engineering is a culmination of the many different techniques and tools I have learned over the last semester. This includes using Eclipse, Ant, Findbugs, Checkstyle, Emma, Hackystat, Hudson, SVN, Wicket, Junit, Google Project Hosting, and something called Java to create DueDates-ulaula 2.0. The project was a team effort from Tyler Wolff, Mari-Lee Flestado, and myself to create a web application with the following features:

• A login page that would make use of an XML file to authenticate users.
• A results page capable of displaying a dynamic list of libraries, based on user data. The page can update and display lists of books due at the users libraries, and can be sorted and filtered.
• An alerts page capable of enabling a process to generate daily email updates for one or more of their libraries.
• Use CSS to create a consistent and pleasant web application.
• Adhere to the three prime directives of open source software engineering: the program accomplishes a useful task, a user can easily install and use the application, and a developer can easily change and extend the functionality of the program.

Our team was able to mostly fulfill these requirements, though there are several areas I would still like to improve in the future. There are "extra credit" features that we were not able to get into, but would be good extensions to implement in the future. It could also use more testing on typical user machines to make sure that the many tools on my computer aren't keeping the program alive even though a user would have problems.

Problems Encountered and Solved

After various administrative tasks, the first problem we encountered was simply deciding between our DueDates 1.2 projects to see which would live on. Though I really like what my own could have been capable of, I did not feel it was quite ready for the task. We end up using Tyler's, DueDates-gold, and it worked out well with minimal changes necessary. This was of course accompanied by the various issues of getting our classpaths set up properly. We also found that as we went along our testing suite became more and more burdensome due to the length of time to complete so some mock-library testing was set up that would be able to exercise most of the code, but skip over the parts that relied on library sites. From there, most issues simply had to do with learning how to properly use wicket features, such as checkboxes, and also various troubles with WicketTester that took some time and team effort to get.

Group Dynamics

Our group worked quite well as a team, though mostly somewhat separately. We met in person just a few times a week to go over planning and design issues, crank out any pressing problems, and get a feel for who would be working on what. We then would be in regular contact through Google Talk. I found it to be generally a better experience than working with only one other person for adding to our breadth of comfort zones and skills. For example, Tyler did almost all of the CSS and layout as he was most skilled and interested that area. I found myself jumping around a lot, perhaps too much, but enjoyed both adding to the code and also keenly looking for problems in the code. More people also meant potentially more conflicts while working on the same code, but luckily the project was of large enough size that this was not too often an issue. I believe we all contributed fairly equally to the project overall, though not necessarily in specific areas. Our Hackystat dev time showed fairly consistent work, especially from Mari. Tyler and I had several days where we didn't work on the project. I know my official time feels a little low, not taking into account my times staring at the code just trying to find an error, or Mari and Tyler's time working outside of eclipse on the wiki guides. And though Mari shows the most dev time, Tyler and I generally built and committed more often. Each one of us has at least one category where we lead by a large margin, which I think shows more of a difference in development style than any overall large difference in effort.

Continuous Integration and Software ICU

During the course of the project we used Hudson and Hackystat to keep continual updates on the health of the project.

This is the final portfolio analysis from Hackystat for the two weeks of development. Overall the health of the project was fairly average, Development and commits took place regularly, though increasingly toward the deadline. Coverage was around 60% through most of the project, but improved near the end, though this was more reactionary and indicated a lack of test driven design, an approach we occasionally used, but often would get forgotten. Our coupling showed a steady rise, but I don't think this was bad considering the larger number of classes necessary compared to previous projects. The statistics are not all-telling, but I did find useful for getting a feel for where the project was at and where it needed to get to.

More to learn

As I continue to grow as a software developer there's definitely some areas I want to improve on. The first is on general design. I want to get better at being able to see just what a class, method, or application is going to need ahead of time so there's less changes to be made later. I also really want to get better at test driven design. I do it in pieces, but too often get sidetracked away from the original intent. However, that which I see I feel is very useful. I also would like to expand in the more social areas of software engineering - specifically improving my leadership skills and also taking more effort to pair program. Many times during this and other projects I would come across (or submit myself) a problem in a commit that I know never would have occurred had I been working directly with that person. Years of working on my own in computer science classes, without many of the tools I now know, and with testing always at the back of my mind rather than the front, makes it difficult to break some of these old habits. However, this project has definitely improved me bit by bit in all of those areas and I look forward to the next leap forward as a software engineer.

Hawaii's high-tech industry: now hiring!

The ICS Industry Presentation

On Thursday, November 20, seven companies local to, or with locations in Hawaii visited UH Manoa to share information about their companies, missions, people, and products. As a student of the university for only about six more months, I'm more motivated than ever to see what Hawaii can use me for, lest I become part of the brain drain to the west coast, which certainly doesn't have as nice weather. Luckily, I was pleasantly surprised by several facets of the presentations and hope to take advantage of the information and networking gleaned from the experience.

Oceanit

Scott from Oceanit started things off with his experiences that lead him to Oceanit. His presentation felt a little fragmented with its terse slides, but was interesting and he is an excited presenter. As with many companies presented, this company feels like a science think-tank, always looking for a great new idea. Also, like many companies, it is closely tied to military contracts which is of interest in today's times with our ongoing conflicts in southwest Asia and a new president and congress coming into power. I spoke with Scott afterwards and he noted that they were mostly looking for people with experience in image processing, which I'm afraid isn't me, but seems like one to keep my eye on.

Alion Sciences

The next presentation was by Darren and Sid of Alion Sciences. Another company with close military ties, they emphasized their experience working on modeling and simulation which was quite interesting. The presentation felt a little disorganized as they tried to share their time and delegate their speaking, but it sounded like an interesting place and they had some really good information to share. It still is impressive to me how close to the rest of the world Hawaii is today, such that meeting with team members across three states and 9000 miles is merely an issue of time zones.

Referentia Systems

From Referentia Systems Inc came a Austin, a recent UH Manoa graduate. Another company looking for great ideas, this was the second company in a row, and not the last, which made me think "Maybe Johnson is right!" Ant, SVN, Junit, test-driven design, etc are no longer complete mysteries to me and I feel like I am developing many of the skills that these companies are looking for. I spoke with Austin afterwards and definitely am going to look into their company some more. Unfortunately, I was "that guy" and managed to leave my freshly printed copies of my resume on my desk that morning!

Datahouse.com

I didn't get a lot of notes for this presentation, or the presenter's name, but I plan to look into it more as database issues are of some interest to me and an important field to experience. The presenter seemed a bit nervous, but knowledgeable of the company.

Camber

Presenting for Camber were David and Kevin who shared information about their First Responder Readiness System. They didn't have a Powerpoint presentation, but were perfectly able to share a lot of information about their company and their experiences with software engineering. Their focus seemed mainly on Ruby on Rails design, which I'm not familiar with, but am sure I can pick up.

Ikouza

I missed the name of the presenter for this presenter, who's company comes out of the Manoa Innovation Center, which I'd heard of, but didn't really know the background of beforehand. While the presentation was interesting itself, the information about the MIC and Act 221 I found to be at least as important to know about.

Concentris

The final presentation was by Larry from Concentris, another MIC related company. They again have military ties, though with a product viable for commercial use. The presentation was well put together, and he even had a sample to pass around. I always find it good to have a solid product at the presentation - that was a good touch.

Thank you to the presenters

My thanks goes out to the presenters who came to see us, and also for the cookies - I was kinda hungry after three hours! I really enjoyed the presentations, and was especially pleased to see that there are companies right here looking for people with the skills that I'm developing. I'm glad I made a good effort to be seen and heard during and after the presentations and hope I left a good impression. A very common thread throughout the presentations was the emphasis on failing is okay! Try, fail, LEARN, repeat, and succeed. That's how great engineers and great products come into existence. They wan't people with good skills, but more importantly a willingness to learn and expand and reapply those skills. I thinking I'm definitely up to the challenge and look forward to proving it very soon.

Show the world how a stack works using wicket.

A stack is a basic structure in the world of computer science. We learn stack implementation early and it pops up (no pun intended) again and again over the years. For the uninitiated, I've created a web application to demonstrate the basics of what a stack does. Implemented using Java and Wicket I present stack-wicket-aricwest. You can download the stack-wicket-aricwest.jar file from the featured downloads. Assuming you have Java 1.6 installed, open a terminal and navigate to the directory you downloaded the jar file too and enter "java -jar stack-wicket-aricwest.jar". This will start the Jetty server. Then open up your favorite browser and navigate to "http://localhost:703/stack-wicket-aricwest/" to begin playing with the stack. You will be able to push (non-empty) strings on to the stack and pop items off the (non-empty) stack. You can also clear the whole stack. A table shows the current contents of the stack.

Challenges

Though a simple application in theory, the challenges of learning a new tool, Wicket, and integrating example and base code into my distribution proved to be a hefty undertaking. All together I probably spent 12-15 hours hacking, debugging, researching, and optimizing. A bit more than expected, but a very rewarding experience. Of course I didn't make it easy on myself: the first thing I did when starting was reconfigure Eclipse to complain about just about every warning possible. This definitely helped me with my code most of the time, though there were a few warnings than I decided to ignore during this project, and some that I turned back off as I felt they did not really apply to my coding standards. Following are some of the challenges I encountered.

Classpath variables: We've gotten a lot of experience with this lately so I only encountered minor problems while getting these set up. Luckily I get a lot of practice because I generally work on two different machines over the course of a project.

PMD hates me: Useful as it is, I was particularly perplexed not by why I was getting warnings, but why the example wicket projects, some of which had the EXACT same code, did not get warnings. I wasn't able to quite figure this out from the pmd build materials, but they are cause for some concern on my part. Specifically in my StackSession where my stack is declared final since I do not actually modify it in that class. But an unmodifiable stack spooks me a bit and I need to figure out the process that allows this to be okay, since my webapp appears to work, or how to deal with this warning like the examples have. Similarly, PMD demands that my Jetty class be made final with a private constructor, which sounds fine... but why does Start.java in my WicketExample04 not get flagged. I'm not sure.

Only one submit per FormTester: This was a bit of a stalling point for me but looking through Google and Wicket in Action revealed that a FormTester is only allowed a single submit. Then it has to be remade. Kind of a pain, but copy-paste isn't hard.

Pick a port, any port: No not that one, it's being used. I definitely need to have a better method for port designation. It also wouldn't hurt, I don't think, for me to be able to specify the port at the command line, which should be a simple fix if needed.

Serialization: Quite simply, I need to review this concept and get a better understanding of why and where it needs to be used, but I was able to get by using examples.

Inner classes: For the most part did not have too much trouble with inner classes, though got caught a couple times by the issue of simply what it does and does not have access to.

Sessions: I'm still a bit fuzzy on these, and whether my application is properly using them. I was able to get separate stacks going using different browsers at the same time, but not with different tabs in the same browser, so not entirely sure if I'm hitting this requirement properly.

Build files: The build files that were cobbled together from various projects had some important differences. My build.xml contained it's own jar target which caused some unexpected results during distribution building. I also changed jar.build.xml to meet the requirements of being able to create the jar file and immediately run it in the same directory. The jar.build.xml actually puts the jar file both in the usual build directory, as well as the base directory. dist.build.xml also cleans up after running jar so that the jar is not including in the distribution.

Cannot pop an empty issues stack

Actually I'm sure there were others too! In any case, these, and dozens of little Wicket bits of learning added up to quite a chunk of time, but a lot of good experience with these tools. Overall, I'm impressed with making a working webapp, but part of me looks at this and thinks web programming can't really be that inefficient, is it?! For example (I think) the stack table gets completely rebuilt each time it changes. Also, it feels a bit dirty having basically my whole program in the StackPage constructor! I was very happy with my testing though. While some was reactionary, I am continuing to make the effort to do test driven design and coding, as I really see how useful it can be in the brief glimmers hiding behind old habits. I also continued to think about version control and continuous integration by creating a repository and diligently trying to always verify (I think I'm addicted to "Build successful") commits and submitting them often.

Due Dates 1.2 - email results and periodic querying now available

The Good

The newest version of Due Dates is now available for public and developer use! Several new changes and additions were made to the system to allow it to output queries to a specified email and also allow the program to sleep in the background and then requery and output results.

Implementing -console was quite trivial. I simply added a boolean flag that would be true if the -console command was encountered and allow printing to go as normal.
Implementing -email was a bit more involved. An email method was created that took an email and smtp server and created a message to send. To create the message, the print method that console was using was rewritten to return a String, rather than printing out. Then that string could be sent in the email message. I had some troubles with setting up the smtp server, not realizing just how that worked. My first test run it went fine, but from then on it wouldn't work, having been flagged as a potential spammer because I was trying to send it through mail.hawaii.edu, rather than my local ISP. After that was figured out, the messages all went through fine. I also had some troubles with the string being created and pmd disagreeing on my style. I ended up including several .append(" \n") with the space included so that pmd would not complain about adding a single character.
The -wakeup option was very satisfying to get working. Initially it had some problems that caused it to create several new DueDate objects each time the timer fired and really showed the weakness of the dependency between the DueDates and Console classes. However some tweaking eliminated the multiple object creating, though it does still perform an extra query in the beginning. Overall I was very excited and happy to see it performing a useful task!

The other thing that I found to be very satisfying is I didn't really have to touch any of the Lender classes. They were self contained enough to not care about the changes and additions going on in DueDates and Console, so that's a good sign!

The Bad

While the basic features were implemented, they are not very well tested. With the current design I found it difficult to test individual methods and classes as much as I would like and this will need some vast improvement. On top of that, my sensor data that I thought was working, apparently is not. It shows very little of my activity in the latest build despite hours of development, and dozens of builds and commits. Unfortunately I didn't realize this until too late to see if I could fix it, since the daily build was done at that point. The daily build was also failing, though the Hudson builds were still usually working. It seems there's a problem with what our test cases expect and what the daily build is doing.

The Ugly

The last week saw a complete breakdown in communication between Daniel Tian and myself. We discussed some design issues a couple times, but we have had disagreement over proper design of the project and tenets of object oriented programming. Busy with other work as well, we did not get much development in for the first several days of the project. I was waiting for him to commit some code changes he had been working on, but they didn't come in for several days, right after I started to begin making changes toward "my direction." Daniel then did some work on the test classes and cleaning up the code structure. Work really didn't begin on the new features until November 16. I began and committed some basic code for the -email method and shortly after Daniel made a slight change to it and that was the last commit I saw. I continued from there to implement the remaining features, working my way around the disagreeable interaction between DueDates and Console class, but not wanting to do a whole new transfer of code to fit my design ideas and cause further confusion of how the code worked. In the end, I got it working and did the wiki updates, but am very unhappy about the design and dependencies between DueDates and Console. It does not feel or look at all elegant and easy to maintain and update, at least to me. We just never were able to solidly hammer out a good design for the project so it's been a continual battle to steer the project in the right direction, and a battle we haven't been facing together. There's still time to get things straight though.

Hackystat and monitoring the "health" of a project

The latest tool being integrated into team orange's development of the Due Dates application is a monitoring tool called Hackystat. It's analogous to a hospital ICU patient monitoring system, but for software development. Various vital signs of a project are regularly recorded through various sensors embedded into developer's IDEs and code repositories. Hackystat can then display trends and latest statistics on the project including statistics on complexity, coupling, testing, lines of code, number of builds, developer time, and others.

assertTrue(this.version.equals(hackystat.getRequirements()))

After a bit of reading up on the material, I started first be setting up my local system to be able to report my usage through my Eclipse IDE, but immediately ran into a rather crippling problem, despite just a few very simple steps. The Hackystat plugin simply would not show up as an option to configure. I readded it several times, a bunch of Eclipse restarts, and a couple computer restarts, and double checked the instructions and my version of Eclipse against the requirements. Later, realizing/told that I was using a older version of Eclipse, 3.3.0, rather than the newer 3.4.0., though Hackystat required only 3.3.0, I installed the upgrade. At that point, Hackystat worked with no problems, so I notified a Hackystat developer about the situation.

Saved again

Having gotten the hard part out of the way, I was able to get start exploring the interaction between my projects and the Hackystat collection tools, which tended to be pretty verbose! At this point my partner had already set up Team Orange's data collection, but I went over it again to make sure everything was in place, which it appeared to be. For the most part the process was smooth but I did get a stuck a couple times with issues in /.hackystat/sensorshell. The first one being, I couldn't figure out where it was. I was a little confused about how it got there, how Hackystat found it, and what I could do about moving it since it was automatically created in a dreaded directory-with-spaces-in-the-name. Then, I seemed to miss the part about setting three variables, but saw the part about if this is your first time you won't need to change anything (else). I was rather perplexed for a time that I could not find the server. Eventually though my partner was able to get me back on track and Hackystat sensors started to pick me up on radar under all of the different build tools.

Our mostly healthy project

After seeing the first build, the project was appearing pretty healthy. Low complexity and coupling, over 80% coverage, though no real work in the last 24 hours. From there, we've gotten to see a few more days of monitoring. Due to some changes with handling, our rather intensive test suite and some design changes, the coverage had dropped off dramatically. Most other stats were healthy, so it's no cause for panic yet, and should pick up soon. I personally haven't had too much opportunity to commit code lately so hoping to get some of that done this weekend, and hammer out some additional details with my partner about our design and testing. I'm optimistic the results will boost our project well into the healthy range, and hopefully no major surgery will be necessary.

Expanding code functionality - introducing Due Dates v1.1!

Having some success with the previous incarnation of the DueDates-orange project, and some tragic failures in our testing, wiki guides, and issue tracking, my project group was tasked with adding some additional functionality to Due Dates. This included a new library to query, as well as the new options -sort and -within that would allow for more customized reporting of results. We also started using a continuous integration tool called Hudson.

Welcome to Hudson, the weather is currently sunny!

Right away Team Orange got together and set about adding our project to Hudson, however, we made just a little checkbox error and ended up spamming about 70 commit logs over the next few hours. Other than that little hiccup, I found Hudson pretty useful for getting a second opinion on my project with each commit. I also enjoyed looking at the other projects and seeing how they were doing, and where other people were running into problems.

I have issues

My first task was to create a list of issues for this iteration of the project. Having learned a bit more about issue management, I went about making a large slew of issues regarding designs we had discussed and each of the new feature tasks, as well as issues for problems that had not been properly raise and resolved in DueDates 1.0. A large number of issues were added, some would end up being invalid, but nearly all were completed. I tried to break them down into small components, however some were more general to account for common wide-ranging commits such as Javadoc updates.

Why is this class doing this?

We did have some trouble pinning down a real solid design for the application. New features and realizations caused several major changes during the course of working on 1.1. Though we've had a pretty good idea on how to implement the project since the beginning, and have worked very hard to make it expandable, there were a lot of cases where we decided certain functionality needed to be moved around. Some times these moves could cause infamous huge commits that would lead to collision problems later.

See you... online

As a team, we weren't able to meet too often unfortunately. Busy schedules and somewhat opposite productive hours (Daniel is bit of a night owl, me not so much) did cause some problems, but we were generally able to iron things out. Always exciting to log in and see what happened while you were out! Despite being fairly productive on the project, I definitely see how much time could be saved working as a pair. There were a number of times when code was committed that would have been caught by the other person immediately if we were pair programming. I personally had several other big projects to work on for school and wasn't able to make commits every day, but rest assured, I certainly thought about Due Dates every day, often making charts and coding on paper in other classes! *hides*

Passing the test

Currently the two biggest issues the project has is proper testing and proper exception handling. Daniel implemented the vast majority of this code and we are both still pondering if they are properly done. The tests take quite a bit of time due to the many log ins to lender systems, but they are fairly thorough. I'm sure we'll come back to these in the future.

Conclusion

Overall, I'm fairly pleased with the DueDates-orange project. It's grown quite large and I know there are improvements to be made, but I think the functionality is pretty good and the design will allow for quick implementation of new features. We've also greatly improved the wiki guides so that the project can be used by other people as well, not just our two-person world. Stay tuned for the latest news and avoid late fees!

DueDates code review #2

It was time for another round of reviews of various DueDates projects in my software engineering course. We had a few problems with the last round of reviews - some difficulties with Google Project Hosting - but learned that some upgrades had just gone in that would make things a bit smoother. Well... I didn't seem to lose any of my comments as I was reviewing DueDates-Gold, but I never seemed to have received a review from Jeho Jung and Yasuyuki Kaneshige of DueDates-Green who we had submitted a request for review of our own project, DueDates-Orange. I'm not sure what happened - whether it was a problem on our end, or Green's, or both, or Google's. I also managed to have left my comments to Gold on revision 39, while Daniel Tian, my partner, ended up with his on revision 40.

In the mean time, I did comment (and I hope it was received) on DueDates-Gold. I had actually previously review this code and was pleased to see a bit of clean up on their part, though I did find a few minor issues and suggestions. Their developer and user guide is very nicely put together, and a good thing, because I had numerous difficulties with the initial developer set up, before I read the instructions! Also I found the use of the ArgsParser to be very interesting and definitely will consider using something prebuilt like that in my own program. The Javadocs also appear to be quite thorough.

For the most part though, I think that their design needs a lot of work. I don't believe it will expand well to the addition of new features and new libraries and lender querying tools. Some methods need to be broken up and separated, while the classes in some areas need to be much more focused and possibly the introduction of some new classes. Their testing also needs to be much more thorough since the emma scores were all in the teens.

Having not received any reviews for my own system, I'm a bit saddened. I know it has a lot of work to be done, and we know what many of the issues are, but another perspective is always very handy. Hopefully next time, as I'm sure there will be!

The impact of project reviews

Having recently finished our first iteration of our DueDates project, Daniel Tian and I set up a frozen copy of the project code and a new wiki page requesting reviews of our project. As classmates in our software engineering class were doing similar, a "review trading" system was implemented. Each person was given three other projects to review and each project would be reviewed by six or seven people in hopes of eliciting a broad spectrum of opinions and corrections.

Getting the most out of a review

In order that the different segments of the code and documentation were adequately reviewed, given that the busy schedules of the reviewers allowed for only a limited amount of time reviewing the project, we broke up the project into several pieces and assigned 2 or 3 of our reviewers to each piece. We knew our reviewers ahead of time, but on reflection, in a more widely known project a request for review would probably not know all of its reviewers identities. In that case I would break up the reviews perhaps by first letter of login name. A-H for one section, I-Q another, and R-Z for the last. This of course is not without weaknesses. What if no one is in the A-H range? What if the people in that range are assigned to review GUI code when they are much better versed with databases or documentation? That issue could be good or bad. These are all issues that should be kept in mind when requesting a review. In any case we broke up the project so that each person should at least attempt to run the project, but then concentrate their review on various pieces that were closely related. The assignments can be seen here.

Though we knew many of our errors before hand, and the review was merely a formality in some ways, our reviewers found several good improvements to be made. Erin Kim made several comments that suggest better ways to identify the correct tables when searching the UH Manoa Library. Creighton Okada and Robin Raqueno added how the use of arguments could be made more general in error reporting, and that some errors were slightly misleading. Daniel Arakaki left a good number of error reports regarding problems and shortcomings of the Javadocs. Anthony Du suggested that the CommandLineParser class possibly be refactored to only deal with the parameter parsing, and not so much the actions associated with the parameters. This is suggestion we definitely have been putting some thought into as well. None of them noted that our project had no unit testing at all. I did not see any comments from Scheller Sanchez however I'm not sure the issue in this case. (edit: I mistakenly said Tyler Wolff did not comment, but forgot that he had emailed me directly.) It could very well be I did not set up the review correctly through Google Projects, as I was a bit confused myself on several parts of the review process, as I'll mention shortly.

My reviews of others

I was charged with reviewing duedates projects silver, gold, and yellow. As is often the case with looking at someone else's code, I waffled between "thoroughly impressed" and "wtf is going on here." Though I had specific assignments to concentrate on, I found that I quickly wanted to review a lot of areas. As each one went along, I applied what I'd found in previous projects to the current one, having already seen common small errors it was quick to find and comment when I saw them again. I found myself often making comments that basically amounted to "you might consider doing it this way as that's how I did it in my project". I made many comments regarding code that I did not feel was as general and easily extended as I felt it should be to account for the growth of the project.

Difficulties with Google Project

While I find it amazing and wonderful that systems like this exist. It is so easy to set up, and of course free, but using the system does not always go as planned. In one case I "lost" some of my review comments somehow and began starting over, only to find them reappear later. I also began receiving emails from other projects regarding other people commenting on the project, apparently because I had started the review process, which again I found strange since I knew I wasn't the first review in that case. Daniel and I also found out as a result of this process, that our discussion group was not set up correctly and we were not receiving emails regarding comments on our own project. All of this does take a bit of practice and getting used to it seems!

Final thoughts

The review process is important for any project, whether it be software engineering or baking a cake. A project needs a variety of opinions and an organized way of collecting them. We're lucky to have tools like Google Project Hosting to help us with these tasks, even if not always as smoothly as wished. Hopefully future reviews, and I know there will be plenty, will be smoother and even more productive.

Introducing the Due Dates application.

Having outgrown stacks and added some new tools to my knowledge base, it is now time to start on a new project that I will be working on for at least the next two months. It is called Due Dates and is an open source Java application that can be used to track the due dates of borrowed items, such as library books. You can read all about the project, download it, and contribute here. Right now the application is fairly simple with online terminal output and access to a single lender, the University of Hawaii at Manoa library system, but in the future I hope to expand it to allow personalized settings, customizable reporting such as email and instant messaging, and a large database of procedures for accessing lenders.

Getting started

The project was started by Daniel Tian and myself. We began by setting up a project hosting site at Google Code and familiarized ourselves with the features and usage of the site. I added some basic issues to the issue tracking system and also began a rough skeleton of my initial program structure. I felt that there should be a static class that outlined the basic features for querying a lender. This LenderQuery class would then be extended for each lender and implement the procedures for querying that lender's website.

First meeting

Daniel and I were able to make quick progress on implementing a semi-functional system at our first meeting, however we sacrificed some of the good software engineering practices we'd been recently learning. We almost immediately set to hacking at our Httpunit example code without really finalizing a design plan and didn't do any unit testing along the way. We did run into a number of problems with setting up our environment to properly recognize all the necessary libraries. However, by the end of our first meeting we managed to commit code that was capable of at least determining if there were books checked out of the UH library. We had also learned a lot about library dependencies and endured some SVN problems, which turned out to be introduced by changes in some of our sample code. After three or four hours of hacking, we only had one commit to show for ourselves, another mistake in our process, and had forgotten (and continued to forget) to label each commit with the relevant issue being addressed. It was a successful first meeting, but definitely made me aware of how hard it is to break old habits and develop good software engineering processes.

Second Meeting

At our second meeting we were using a different computer and fixed some of the issues with dependencies and SVN that had not previously been noticed. We also discussed some of the problems with the approach we used in previous meeting and introduced some new issues to the issue tracking system. After a short meeting we were able to run the application with increased success, having implemented the ability to list the books on loan (which I had checked out that morning). Armed with a false sense of accomplishment, we made the grave mistake of waiting several more days before working on the project again.

Some time later

Yesterday, I began making some modifications and refactoring the code. I added a BorrowedItem class to the project and cleared up some of the class and method names to be more descriptive of their tasks. I practiced doing small commits often, rather than large, multiple file changs, however almost universally forgot to address which issue I was working on in my commit message. The issues needed to be broken up and though out better anyways, but I simply always forgot to add the "Issue-N" header. And I also completely forgot one commit message!

Later on that evening Daniel and I got together online and discussed what we needed to finish and change. He did some more code modification to try to improve toward our goal of making the program as extensible as possible. He also helped me with some mistakes I had made with designating the library dependencies. In the meantime I worked up a couple of wiki guides, one for Due Dates users, and another for potential developers.

Lessons and Experiences

Overall this was a very good experience and really demonstrates the power tools like SVN and issue tracking. However it also showed me that I have a long way to go toward becoming an effective software engineer and using those tools wisely. SVN makes it very easy to work independently from the rest of your team, but it is no substitute for good communication. The main issues I believe I need to work on are effective design, creating unit tests, and getting a better understanding of how to properly set up libraries so that another user can quickly begin to use or modify my code. Also, quick progress in the beginning doesn't I should reward myself with three days off! Overall though it has been a positive and rewarding experience.

I see more collaboration in my future

To add a little experience to my work with Google project hosting, as discussed in my previous entry, John Zhou has reciprocated the sharing of code and set up his own stack project page. Similar to my work on the stack-johnson project, I was able to easily set up stack-johnzhou on my workspace and do some modifications. His code seemed clean and tested. The output location was correct as well. I just added the simple clear method to the stack class and ran verify. Of course Emma coverage dropped without some testing, but it is otherwise still able to compile and verify.

Overall, most of the tools used in regards to SVN were very simple to set up and use. However, I'm sure there's much more to be discovered about them that help prove that a tool can be easy to use, yet immensely powerful with some experience. The difference between his method of collaborative programming and the process I used with Arthur Shum on our CodeRuler project are immediately obvious and profound. Arthur and I's manual sharing and combining of code for that was tedious and time consuming. Also, during this project, John showed me how to use ant tools in Eclipse, rather than the command line, just by right clicking an xml file and selecting Run As... Ant. I felt downright silly for not having found that earlier!

Making a collaborate effort.

A programmer working on his or her own can only accomplish so much before the limitations of time and talent begin to make the completion of a large project infeasible. To succeed in the modern programming environment requires collaboration between several or even thousands of individuals. Furthermore, for such a collaboration to be successful it is necessary for code to be shared, updated, and shared again using reliable and efficient methods. One way of accomplishing this is through SVN or subversion control. By creating a central depository of code and implementing safe ways of distributing code and tracking changes, it is possible for a large number of people to work together on a single system.

Google: a generous overlord

One source for a framework for SVN is through Google project hosting. Anyone can, for free, create and manage their own open-source project using this site. To help my classmates and I get comfortable with the subversion process, my software engineering professor set up a project containing the familiar Stack project. In order to easily be able to check out and update files from this repository, I downloaded a tool called TortoiseSVN. After a quick install and reboot (which I always hate because it can really break the productive process) I was able to create my own private space to work on the code downloaded from the Google project page.

Making my mark

Having downloaded the code, which at this point had been modified by most of the students in the class, I was worried there wouldn't be much for me to accomplish. However, I found that Stack still did not have a simple clear method to empty it, so I added that. What's more, the project was set to output to a /build/classes directory, rather than /bin, an issue that had previously caused strange problems in Eclipse. Having fixed those issues, I was able to quickly (after finding the cryptic password again which for some reason was not saving) upload the code along with a message about my changes. During my work with this project it also appeared to have acquired an empty and mysterious extra stack-johnson directory, which I deleted.

An offering to the world

Having successfully modified and updated code from a Google project, it was time for me to create my own project and to help test it, put it at the mercy of a fellow student, John Zhou, and the world, if it is interested. My personal latest iteration is no longer available only from a archaic zip-file in one of my other blog posts, but can be also found here. I also created a discussion group here. This whole online project world will also make it easier for me to work not just with others, but with myself in different locations, on different computers, which is often what I do. I definitely look forward to adding another layer of control and organization to my lessons on software engineering.

An incomplete Emma example

I mentioned in my previous entry that Emma is a good tool to see the coverage of JUnit tests. However, it is only a guideline, not an absolute measure of good testing. Simply being able to claim 100% coverage does not mean that all possible states in the program have been tested or that the tests were formulated correctly.

As a simple, if rather contrived example, this distribution contains an artificial error in the pop method of the stack class. By simply inserting a modulo four into part of the code it operates normally for pops when the stack is of small size, and Emma continues to report 100% coverage. While pushing three objects on the stack and popping it once is fine, pushing five objects on to the stack and popping once, which could be considered an equivalent class of testing, it might fail, depending on the pushed object. It will fail if the showNewError test in the TestStack class is not ignored, but the other previous test suite did not catch this error.

Though it is a rather contrived bug, it shows how important thorough testing is and that a programmer cannot simply rely on coverage tools to indicate good testing has been done. Such tools can tell you that tests are missing, but does not tell you if you're really done testing. That requires 110% effort and a report of 500% coverage or more!

Coverage testing with Emma

Among the tools I downloaded and set up while working on my stack testing (links to the files I talk about in this post can be found there) is one called EMMA. It is a coverage tool which goes together with JUnit testing and reports to the user how much of the overall code was actually seen by the test cases. Here is a part of the sample output from EMMA:



It shows, for example, that only 50% of the methods in ClearStack were actually tested, 39% of the code blocks, and 50% of the lines in the class. EMMA doesn't give the user any information about how useful the actual testing was but, as a minimum, it does show that there is much more testing to be done.

Working toward 100%

I started out with the Stack.java testing and simply added tests for the toString method and a test of the top method on an empty stack. This brought my EMMA report up to 100% complete for the Stack class simply by adding two rather trivial tests. This just means I covered the baseline. I know that at least one test was done on all portions of the code.

From there I started to improve on the TestClearStack class. One of the first problems I ran into was not with the ClearStack class but in the testing class. It did not test any methods that could potentially throw exceptions, and didn't have a throws EmptyStackException in its declaration so Eclipse immediately started to complain when I started adding additional test code. The addition of a few tests for isEmpty and getTop caused Eclipse to become upset that it could not resolve ClearStack and EmptyStackExceptions to a type. This error seemed very odd to me, as JUnit and EMMA were still running just fine and my coverage was improving. I was pleasantly surprised to find that EMMA reminded me to check for both true and false conditions on the isEmpty method. However, for the other errors, I decided to run ant -f verify.build.xml and discovered that a slew of new CheckStyle errors had crept into parts of my code which I didn't even think I'd touched. Slightly baffled, I corrected those errors, made Eclipse and CheckStyle happy, and was eventually able to verify the entire project. Here is a link to the final distribution.

Conclusion

Overall the whole experience with EMMA was minimal. I didn't find any new coding errors in the Stack and ClearStack classes while striving toward 100% coverage. The toString method seems odd to me though. I would expect it would want to print out as "[1, 2, 3]" like similar collections, as opposed to "[Stack [1, 2, 3]]", but the documentation did not really mention the expected output so I left it as is. Most of the learning was in dealing with a number of issues with CheckStyle and Eclipse, plus a little more experience with setting up my project workspace correctly. For example, my first distribution build ended up being called "stack-aricwest-6.0.23-6.0.25" simply because of the way I imported my previous code. Even though I didn't find EMMA particularly useful at this point, I definitely see the potential for finding untested code in larger projects. One must remember that it is only there (from my limited experience) to identify a bare minimum of testing and doesn't indicate that a project has been thoroughly tested simply because it reached 100% coverage.

More lessons from CodeRuler and QA tools

Having had my first taste of Java's available quality assurance tools, it was time to compare it to some of my own code. Having learned quite a lot about improving my coding style and efficiency during the CodeRuler project, it was time to see what automated tools could find. I knew about many of the poor areas in the code due to ignorance or lack of time and proper planning, though the project itself ran quite well. Along with what I learned while working with my project partner, Arthur Shum, I also got the input of two classmates, Robin Raqueno and Ronn Reeves. So I didn't expect too many surprises by running PMD, Checkstyle, and FindBugs on the MyRuler class I helped write.

FindBugs actually reported no errors which at first made me ask myself, "What does FindBugs do again?" It just seems nothing quite fit what it was looking for. That or it wasn't looking in the right place, though the report seemed to be using the correct directory.

Checkstlye returned a huge number of errors, whoever most of this was due to provided IBM code, and only thirteen in my team's contribution. Ten were simply code lines over 100 characters, two were very small white space violations, and the last was a import statement using .* that was included in the original code.

The final check, PMD, was by far the most informative of the three. It found 23 violations, which were significantly more interesting than those found by Checkstyle. Though I was aware of many of these violations either on my own, or from other human reviews, PMD was very thorough and informative. It's suggestions for variable usage were most common. Other mistakes included improper use of comparison operators and not using the more abstract collection interfaces instead of TreeMap and ArrayList.

In the end, these tools are very good at spotting large numbers of errors where human eyes might glaze over, but they need to work as a supplement to peer reviews. Many sections of overall poor code design are quickly noticed by reviewers that automated quality assurance did not bother to note. It was fine by machine readable standards, but most humans would consider many sections of the MyRuler code to be of poor readability. We should use all the tools available to us to write and test the best possible code, both computerized tools and people at every step of the development process from programmer to user.

It's not a bug; it's an undocumented feature.

One of the reasons that Java has become such a popular and widespread programming language since its introduction doesn't actually come from the features of the language itself. Instead, some of its growing usage can be attributed to the vast number of third party tools that are available to make the Java program development process smoother, more efficient, and with less undocumented features. I've recently started familiarizing myself with some of these Java-oriented tools such as the Eclipse platform, Ant, PMD, FindBugs, Checkstyle, and JUnit. Even with just a small amount of experience, I've already starting to think about how I have been able to get along without such tools and guidelines.

How do they stack up?

After downloading and unpacking Emma, JUnit, Checkstyle, PMD, FindBugs, and SCLC the next step is inserting them into the environmental variables listing of my computer so that Ant will know where to look while it is building your project. With that finished I downloaded a test project supplied by my software engineering professor, Dr. Johnson. It's a simple stack class and test cases which are purposefully broken in various ways. Using these new tools, I plan to get the stack class into both a working as well as efficient and consistent system. By running Ant for each of the build configuration files I find that each except "ant -f verify.build.xml" are successful. At first I think that means each one found no errors in the stack project. However, successful means that PMD, Checkstyle, etc. didn't have any problems running - all my paths were correct - however there were plenty errors to be dealt with and generally nice HTML documents to glean over.

Results and repairs

• Checkstyle found two violations of our style standards: a misplaced curly brace and a missing javadoc annotation in stack.java. Both easily fixed.
• FindBugs reported several places that could be changed to be more efficient by changing Integer constructors to Integer.valueOf methods.
• JUnit found the first true errors in the logic, at this point just in the test cases. Code was corrected to reflect a proper pop order of 3, 2, 1. At this point I also caught a typo in the test cases which had no effect on performance, but wasn't caught by other tools. However upon rerunning the JUnit test I still was finding errors and it took some time to discover that the test cases weren't just written wrong, the stack class itself was incorrectly coded in the top function, which was actually looking at the bottom of the stack.
• PMD recommended setting some private variables to final and simplifying a return to get rid of an unnecessary temporary variable. It also reminded me to abstract away from unneeded details such as using a List interface even though an ArrayList is being implemented. The final tricky part was an empty catch block. According to my Elements of Java Style book I could simply use a e.printStackTrace, however Dr. Johnson recommended eliminating the try/catch entirely. I implemented this change as well as cleaned up the unnecessary import of the Assert.fail from JUnit.

At this point I gave "ant -f verify.build.xml" a try and behold: my stack distribution.

Overall it was a good experience with some snags but not too much frustration. The various tasks all seemed to have worked in the end. This is of course only a small part of Ant and the associated build tools. I only had to run them, not really understand them, let alone write them, yet. However I got a good appreciation of what they are capable. I expect it to be a steep learning curve, but see the potential for time saved in the long run. I also expect to be able to apply some of the concepts to other non-Java projects when possible.

Review of another CodeRuler implementation

In an effort to further familiarize myself with the coding practices that I will be using throughout my Software Engineering course (and beyond!), I have been tasked with reviewing one of the other CodeRuler projects submitted to the class. I will be reviewing the MyRuler.java class of Tyler Wolff and Daniel Tian. I downloaded the zip file containing the class and its Javadocs from this link on September 10, but had some difficulties with the zip file and was unable to properly unzip the Javadocs due to a corrupted file. I wasn't able to determine the cause of this error, but was able to extract the MyRuler.java file without troubles.

During our Wednesday class period I was very excited to see this implementation in action as Tyler and Daniel's blogs showed it to be very successful in their tests. Unfortunately it seemed to have encountered a problem during the demonstration and failed to successfully perform any moves. However testing on my own computer was fine and it was perfectly capable of performing up to the standards claimed. In fact it is the only of the implementations I saw that could compete and quite possibly surpass the implementation of Arthur Shum and myself.

Despite not being able to convince my computer to open the javadocs, they were luckily available right in the code and appeared to be thorough. The structure and methods used in the class made sense. Each method was dedicated to a specific task and each well explained. The orderSubjects method is simple and is able to delegate the complex tasks of the implementation to other methods but outlines the workings of the class. The inline comments within various methods help with understanding the logic of the program quickly, though some were redundant. Since many of the variables names consist of three or four words, the code for certain tasks get a bit unwieldy and a number of lines cross well over 100 characters in length. However the logic is quite solid and it's performance outside of the class demonstration is superb.

For the most part the formatting and style of the code is quite close to what is put forth primarily by The Elements of Java Style by Allan Vermeulen, et al. The following chart demonstrates the lapses in proper coding style I observed.

File	Lines	Violation	Comments
MyRuler.java	1		Use of * in import.
MyRuler.java	16-28		Possibly excessive HTML code in javadoc comments.
MyRuler.java	7, 60, 183, *	EJS-49	Omit subject in describing methods.
MyRuler.java	54		Explain empty method.
MyRuler.java	82, 123, 130, *	EJS-7	Closing bracket not on its own line.
MyRuler.java	87		Possibly confusing end-line comment. End of if? or end of else?
MyRuler.java	79, 84, 90	EJS-7	No space between for and (.
MyRuler.java	109		No period ending description.
MyRuler.java	120, 121, 127, *		Four space indent instead of two.
MyRuler.java	120, 127, 134, *	EJS-61	Unnecessary end-line comments.
MyRuler.java	210		May want to use parentheses to seperate logic operations.
MyRuler.java	211, 212, 215, *		Way over 100 characters in these lines.
MyRuler.java	239, 365, 427, *	EJS-7	Beginning bracket for else statement should be on same line as else.

The errors found do not have a huge impact on the readability and reliability of the program. Most can be quickly repaired now that all of the members of the class are more aware of the strict style we should to be adhering to. There is ample commenting of the code, and perhaps too much in several areas where they do not really offer any additional information. Overall though it is an impressive strategy and implemented relatively cleanly and efficiently.

Lessons learned from CodeRuler

Intro to CodeRuler

As a bit of a warm-up exercise to get my software engineering class acquainted and reacquainted with Java and Eclipse we were recently assigned to work on a project using the CodeRuler plug-in from IBM. The plug-in contains a fairly complete framework of Java classes that implement a simple game of knights, peasants, castles, and their rulers. World domination is, of course, the goal. Everything is in place except for the implementation of a strategy. This is where my classmates and I come in. So my partner, Arthur Shum, and I set out to create an AI that would give the provided computer strategies a solid challenge and then later this week we set out to conquer the avatars of our classmates as well.

Teamwork

Arthur and myself started off with a slightly disorganized approach to the project. Initially we just wanted to play around and get familiar with the tools available in the CodeRuler interfaces. From there we individually progressed to various simple strategies and incrementally improved certain ideas and tossed out those that didn't work. Eventually I dedicated myself to programming the behavior of the peasants while he worked on strategy for the knights. I'm proud of my little peasants, but I must admit, his knights are quite impressive and I felt as though I may not have contributed as much to the team. Nevertheless, we worked well together despite not meeting up in person. Communication was done online; we would simply email changes to the code back and forth as we discussed strategies, problems, and solutions on gtalk. Eventually, though we both have numerous ideas that we would like to implement, we got to a point where our strategy was quite successful against the provided computers. We feel it will also be competitive against other teams in the class.

Results

Here are some of the results versus the provided AIs.

	aricwest-atshum	Migrate Ruler
Test 1	829	0
Test 2	822	0
Test 3	827	0
		Gang Up Ruler
Test 1	804	48
Test 2	792	100
Test 3	779	112
		Split Up Ruler
Test 1	805	64
Test 2	827	14
Test 3	801	44

You can find the product of our programming efforts, as well as associated javadocs, here.

Strategy

The basic strategy can be broken up into three parts - one for each unit.

First we have the knights. Arthur came up with a two-team knight approach that sends a small team of knights to capture castles, while the large team goes for enemy knights. To help protect our own castles, the large team prioritizes knights that are closer to our castles. Once the large team wipes out all the knights then it will work on capturing castles, then capturing peasants. The small team instead captures castles, then goes after peasants, then knights. Of course, if either team comes across something else in their path to their main objective they won't hesitate to capture that too. There are some slight variations, but this is the essence of the knight strategy.

Next we have the peasant strategy which I implemented. At first it looks like Random Ruler, and it certainly started out that way. However our peasants are a bit more aware of their surroundings. A peasants basic strategy is to examine the squares around him and randomly pick one that has not been claimed by our team, and is otherwise unoccupied and available. This on its own gives the peasants a quick and rapid start. At some point a peasant will likely become surrounded by land it or another peasant has already captured for our team. At this point our peasant will start moving toward an enemy peasant, since that area will obviously have land belonging to the enemy team. It is also programmed in such a way that our peasants will divide up and chase after different enemy peasants, not cluster after one. Also, if no peasants are available to chase, it will move toward enemy castles. These chase behaviors only last as long as there's no new land to capture. At any point there is new land to capture, the peasants continue with the prime objective - adjacent land. Unfortunately the peasants lack the code to help them evade enemy knights.

The third part of the strategy is simply whether castles should produce knights or peasants. Arthur came up with a simple and effective strategy. Our castles produce knights until we have 50% more than there are enemy knights, then switches to peasants for as long as the 50% more rule holds. A basic overpower-the-enemy strategy, which works thanks to the efficient gathering of land by our peasants, so more are not needed immediately.

Lessons Learned

Over all it was a decent warm-up to Java and Eclipse, both of which I do not have any recent or extensive experience with. However, I think that I didn't take advantage of many features of both that might have made the process easier. Part of it is just not realizing what is available so I work with things I know and understand without branching out enough. I definitely enjoyed using some of the features that had been discussed in our recent software engineering class including the for each and some of basic work with Collections. Working with a partner worked out well and favorably despite never actually meeting in person. In the future I hope to avoid some of the difficulties of emailing our code changes back and forth to merge our additions and changes. An SVN type system would definitely be preferable, but this was a small enough project as to not be a large issue. Our end product is competitive, but I definitely feel it could use some refining in several areas. I would like to add some knight evasion code to the peasant AI. Also there have been time out issues when large numbers of units are in play. Some code could be added to tone down movement when the time out limit is being approached. I'm sure there are also some code that could be more efficient which would help keep the processing time down. Overall though I am pleased with the results and process of this experience.

Assignment 4: FizzBuzz

/* Aric West
 * ICS 413
 * August 27, 2008
 * Assignment 4
 * FizzBuzz.java
 * 
 * This program prints out numbers from 1 to 100, however numbers that are evenly
 * divisible by 3 print Fizz instead, those evenly divisible by 5 print
 * Buzz instead, and if divisible by both, FizzBuzz is printed. 
 */

public class FizzBuzz {
  public static void main(String[] args) {
    for (int i = 1; i <= 100; i++) {
      System.out.println(getFizzBuzz(i));
    }
  }
  
  public static String getFizzBuzz(int number) {
    if (number % 3 == 0 && number % 5 == 0)
      return "FizzBuzz";
    if (number % 3 == 0)
      return "Fizz";
    if (number % 5 == 0)
      return "Buzz";
    return Integer.toString(number);
  }
}

Total elapsed time from opening Eclipse until starting this blog post: 21 minutes. However, I did manage to distract myself on several occasions and could have easily finished in half of that time. While I did not take notes during our first class, I liked the style presented during the lecture and my code I believe resembles that. Java is definitely not my most comfortable language, but I have a reasonable appreciation for the beauty and style of object oriented languages and do not believe I'll have too much trouble getting along. I had no errors when I ran the code the first time as Eclipse quickly pointed out the little things I'd forgotten. The most difficult part, which required 10 seconds on google, was converting the integer to a string. I didn't encounter any problems with Eclipse, though I had already previously installed it on my computer.

In our brief time in this class, and in regards to this assignment the most important thing I think I've been reminded of is that there's almost always a different way to solve a given problem, and quite likely there's also a better way... maybe!