I hope to see you there!

Note: I'm keeping the old content here.

What sets it apart from other such tool IMO is the UI and speed/quality of search. I especially like the ability to see who's sending the most mails to a list and the nice syntax coloring display of emails (in addition to the thread view). Another nice feature is that emails are indexed a few seconds after they have been received by the list (compared to several hours with other tools). I love it :)

My slides are available online.

If you're interested to meet up register on the Facebook event.

We have also organized a joint XWiki and Maven tech meetup on the 9th of October, at Terracotta. Indeed Terracotta has been extermely kind not only to lend us a room but also to cater for some pizzas and snacks :) Thanks Terraccotta!

The address is:

Terracotta Inc.
650 Townsend St. Suite 325
San Francisco, CA 94103 USA
+1 415 738 4059

Jason Van Zyl, creator of Maven will be there too!

I hope to see a lot of you there!

I'm also happy to report that XWiki (the company and open source project I work for) is a Sonatype Partner. I'd love to see Maven products built on top of XWiki. I think XWiki's flexibility makes it an ideal platform for building those products and at the same time empowering them with collaborative features. In the near future I'll work closely with Jason and his team to integrate XWiki and Maven and hopefully we'll see some cool things out of this.

Well done guys and long live Sonatype!

The conference was very well organized and the training room was superb. Thanks Vincent for setting this up.

I've made my slides available here.

I'm extremely excited on several accounts:

• I'll be coming back to my first love which is software development. Even though I've continued doing development over all those years it was mostly during my free time, working on open source projects (mostly Cactus, Maven and Cargo). I'm glad to be able to bring software development in my day time job.
• I've spent the last 9 years doing consulting (IT Architecture consulting and then Offshore software development consulting). I was keen to build a product and focus on that single job. I'll still do varied jobs of course but it'll be centered around a product.
• I strongly believe wikis are only just entering the general market. They were niche tools so far but I have the feeling they're going to get more mainstream in the coming years. Of course XWiki is not just a simple wiki, it's actually a second generation wiki, on which collaborative web application can be developed (in short, take almost any "Web 2.0 application" - whatever it means! - and XWiki is a good candidate for building it).

I'm now off to Javapolis 2006 where I'll be manning the XWiki booth with Ludovic and Guillaume. Come and say hello to us!

We were lucky to have Jason Van Zyl talk about new Maven2 stuff and especially the Repository Manager and the maven.org development platform. Emmanuel Venisse has done a presentation on Continuum, Fabrice Bellingard has presented a return of experience of implementing Maven in a large company and I have done a quick presentation of new quality features in Maven2. See Jean-Laurent's blog entry for more details.

This tool could be based on Clirr or JDiff for example. It would also have an option to fail the build if there are new methods without a @since tag.

Do you know if such a tool exists?

Here's a method we had (it expands a JAR file):

    public void expandToPath(String path) throws IOException
    {
         File workDir = new File(path);
         JarInputStream inputStream = getContentAsStream();
         
         byte[] buffer = new byte[40960];
         
         ZipEntry entry;
         while ((entry = inputStream.getNextEntry()) != null)
         {
              String entryName = entry.getName();
              entryName = entryName.replace('/', File.separatorChar);
              
              String outFileName = workDir.getPath() + File.separator + entryName;
              File outFile = new File(outFileName);
              
              if (outFileName.endsWith("/") || outFileName.endsWith("\\"))
              {
                   outFile.mkdirs();
               }
              else
              {
                   if (!outFile.getParentFile().exists())
                   {
                        outFile.getParentFile().mkdirs();
                    }
                   
                   if (!outFile.exists())
                   {
                        outFile.createNewFile();
                    }
                   
                   FileOutputStream out = new FileOutputStream(outFile);
                   int read;
                   while ((read = inputStream.read(buffer)) > 0)
                   {
                        out.write(buffer, 0, read);
                    }
                   
                   out.close();
               }
          }
         inputStream.close();
     }

Here's how I've transformed the method by removing all File operations and instead introducing a FileHandler interface with the following methods, equivalent to the File ones:

• append(URI, String): appends a suffix to a URI
• mkdirs(URI): create directories for the URI
• exists(URI): return true if the URI exists
• createFile(URI): create a file
• getOutputStream(URI): get an output stream for the passed URI

    public void expandToPath(URI path) throws IOException
    {
         JarInputStream inputStream = getContentAsStream();
 
         byte[] buffer = new byte[40960];
 
         ZipEntry entry;
         while ((entry = inputStream.getNextEntry()) != null)
         {
              String entryName = entry.getName();
  
              URI outFile = getFileHandler().append(path, entryName);
  
              if (outFile.toString().endsWith("/"))
              {
                   getFileHandler().mkdirs(outFile);
               }
              else
              {
                   if (!getFileHandler().exists(getFileHandler().getParent(outFile)))
                   {
                        getFileHandler().mkdirs(getFileHandler().getParent(outFile));
                    }
   
                   if (!getFileHandler().exists(outFile))
                   {
                        getFileHandler().createFile(outFile);
                    }
   
                   OutputStream out = getFileHandler().getOutputStream(outFile);
                   int read;
                   while ((read = inputStream.read(buffer)) > 0)
                   {
                        out.write(buffer, 0, read);
                    }
   
                   out.close();
               }
          }
         inputStream.close();
     }

The interesting part comes now. Because it was a bit hard to create a unit test for the original expandToPath method nobody had done it. It would have involved passing a test JAR but more difficult it would have involved passing a target directory where the JAR would be expanded. This is not easy as the location of this target dir would depend from where the tests is executed and making it work seamlessly from both a build tool and from your IDE is not trivial. Here comes VFS to help us. By implementing the FileHandler interface using VFS, we can now write the following unit test:

    public void testExpandToPath() throws Exception
    {
         URI jarURI = new URI("ram:///test.jar");
 
         FileObject testJar = VFS.getManager().resolveFile(jarURI.toString());
         ZipOutputStream zos = new ZipOutputStream(testJar.getContent().getOutputStream());
         ZipEntry zipEntry = new ZipEntry("rootResource.txt");
         zos.putNextEntry(zipEntry);
         zos.write("Some content".getBytes());
         zos.closeEntry();
         zos.close();
 
         DefaultJarArchive jarArchive = new DefaultJarArchive(jarURI);
         jarArchive.setFileHandler(new VFSFileHandler());
 
         jarArchive.expandToPath(new URI("ram:///test"));
 
         // Verify that the rootResource.txt file has been correctly expanded
         FileObject rootResource = VFS.getManager().resolveFile("ram:///test/rootResource.txt");
         assertTrue(rootResource.exists());
     }

Notice the use of the "ram:" URI scheme. This one of the many filesystems supported by VFS and it means that all file operations will happen in a virtual file system in memory. Also note that VFS doesn't currently support creating Zip files so we're using the JDK's ZipOutputStream API. The nice thing is that as this test operates in memory there's no need to define a target location on the file system.

The other nice thing is that by introducing VFS to this expandToPath() method it's now possible to expand a JAR to any file system supported by VFS. We could thus expand to a FTP server, to a WebDAV repository, to an HTTP URL, to a remote machine using SSH, etc. All this without changing a line to our code. Nice isn't it?

IntelliJ IDEA has revolutioned the IDE landscape by adding "intelligence" to IDEs. A few days ago I did a thought experiment by asking myself the following question "how feasible would it be to build a project without knowing any meta-data about it?". In other words, is it possible for a build tool to be intelligent enough to build a project without build files nor POMs. Said differently, is it possible to figure out a project's POM automatically? Let's review some required typical meta-data information and see how they could be guessed.

Source locations

It is possible to guess where sources are by looking for *.java files (for Java projects - The same applies for other project types). Now we still need to differentiate main sources from test sources but that's also relatively easy to do. We can check for classes extending JUnit's TestCase for example or the TestNG equivalent, or any other well-known testing framework.

Note: An interesting thing here is that to be intelligent we'd need the help of the community to add new rules to the discovery process. For example imagine that a new testing framework appears; we'd need to add it to the Test Discovery Rules. Thus, this type of intelligent build system would need to rely a lot on the community and thus would need to get its data from an online repository that could be edited by the community.

Dependencies

How do we detect project dependencies? One relatively way is to parse the sources that we have found above and find all external imports. Then query ibiblio to find matching package names (this information is present in Maven POMs on ibiblio). Now for guessing the version, there's no easy magic. A first approach would be to get the latest released version of the dependencies we've found.

Project type

Project types can easily be guessed by looking at some files. For example if a web.xml file is present then it's a WAR project, if an application.xml one is found then it's an EAR project, if a jnlp file is found then it's a JNLP project, etc.

SCM

SCM can easily be guessed by looking for special files on the filesystem of the project. For example we would look for .cvs directories for SCV and for .svn files for Subversion, etc

Developers

Once we got the SCM URL we can then query the SCM to get the list of all developers.

Project name

The project name could be the name of the top level directory and the version could be set arbitrarily to 1.0. Actually we could even check ibiblio to see if the project is already on ibiblio, get the latest version there and increase the minor number by one as a first order guess. Another strategy would be to query the SCM and look for tags and deduce existing versions by parsing those tags (there are some usual conventions for naming tags so it should be possible to make a good guess).

Modules and artifacts

Discovering the different modules of a project is probably one of the hardest thing to do. If you look at different projects in the wild I believe there are not that many directory structures out there. Maybe 10-15. Thus it should be possible to register knowledge of these structures and let the tool discover which ones matches the closest with the project at hand. This would also allow to deduce the different artifacts that have to be generated. Of course it won't be perfect as there are projects which generate several artifacts and which may be in the same module. Again it's a question of doing 80% of the job and leaving 20% to be done manually.

Additional information

Of course, the information found above are just guesses. In most cases they could be correct but of course we would need to offer a way for the user to edit them and to add any missing information.

Conclusion

I believe it should be possible to create such an intelligent meta-build project which could be used to generate files for one of the existing build system such as Maven, Ant, etc. For example it could create an internal POM file on which Maven could then be executed to produce the build results. At a minimum such a tool could be used to convert existing projects to Maven. I wonder how intelligent it could be but I guess it could go pretty far.

Disclaimer: Of course, such a tool would be bad from a conventions stand point. One of the great strength of Maven has been to standardize the directory structure of projects. I can go to any Maven project and I know exactly where stuff will, what will be generated, etc.

Are there other information which you think could be guessed automatically? Can you think of better algorithms to guess some of the information shown above?

"Better Builds with Maven”"

• is a comprehensive 'How-to' guide for use with Maven 2.0.4 and later
• is available free of charge and includes supporting sample code
• covers how to use Maven 2.0 to better manage the build, test and release cycles associated with software development
• is written for beginning and intermediate Maven users
• is authored by Maven experts
• Jason Van Zyl, Chief Architect and founder of Maven
• Vincent Massol, author of "Maven: a Developers Notebook"
• with chapters and key content and code contributions from leading Apache Software Foundation Maven Project members: Brett Porter, John Casey and Carlos Sanchez.
• is published by Mergere, Inc

Content Includes:

• An introduction to Maven 2.0
• Creating, compiling and packaging your first project
• Best practices and real-world examples
• Creating J2EE builds and using J2EE models
• Extending builds through plugins
• Monitoring source code, testing, dependencies and releases
• Leveraging repositories, Continuum for continuous integration and transitive dependencies
• Converting existing Ant builds to Maven

Download a free copy at http://library.mergere.com.

I hope this book will help boost Maven2's adoption.

There were 2 main goals for this conference:

• Reduce "FUD" spread by influential non-MS technology users by showing them what MS is actually working on. It's harder to spread FUD when you know the details...
• Get feedback from competitive technology users to improve MS products/technologies

I applaud MS for having the foresight (and the money) to do this. It takes some vision and courage to set this up and allow everyone to blog freely about it.

Here are the topics we were shown during these 3 days:

• Scripting and dynamic languages by Jim Hugunin
• .NET CLR 2.0 Reliability by Allesandro Catorcini
• GotDotNet and Communities by Korby Parnell
• Open Source at Microsoft by Bill Hilf
• Microsoft Research by Rick Rashid
• WCF & WS-* by Don Box
• Internet Explorer by Dean Hachamovitch
• LINQ/C# by Luca Bolognese, Anders Hejlsberg
• Executive General Session by Sanjay Parthasarathy
• XBOX Extensibility by Brian Keller
• WPF – next generation UI by Chris Anderson
• Windows Server “Longhorn” as an App Server by Doug Purdy
• ASP.NET/IIS7/Atlas by Scott Guthrie
• InfoCard by Mike Jones
• Windows Mobile/Embedded by Mike Hall and David Karle
• eScience – the Next Decade. Lessons learned and the path forward from TerraServer, SkyServer and bio-informatics by Jim Gray
• Software Factories by Jack Greenfield

What was the outcome? Was it effective?

It was certainly good to be invited (many thanks to Steve for that). The presentations were of mixed quality and I felt that the topics were too broad. There were some that were of interest to me but lots of others were not in my area of expertise/interest. I've also felt that there were not enough participation to meet the original goal defined by MS.

I'd like to believe Microsoft was serious about the feedback we gave them but I'm not sure how much I can believe this... The reason I have some doubt is because the sessions were not meant to gather feedback but rather to explain how things are done at Microsoft. If the goal of this conference is to get feedback then I think the format of the presentations could be much improved. Here are some ideas for next year (in case there's a MTS07):

• Mix formal presentations with round tables sessions. Get 10 persons per table with each table having a Microsoft coach to drive discussions. Have different topics per table and let attendees pick the topics that interest them.
• Use the SPA conference session formats as exemples. This idea is to have more workhop sessions than formal presentations to get everyone's participation and to get deliverables as part of the session's outputs.
• Feedback from everyone was provided at different point in time and to different Microsoft employees. The feedback was received verbally and there's currently no guarantee that the feedback will be remembered and acted upon. I'd suggest to have a large white board that is used throught the conference to record all ideas for improvement. This will also stimulate feedbacks.
• Report on all feedback submitted during the previous conference at the begining on the conference to show what impact the feedback has had.

Note that I have received an email from MS pointing to survey where MS is asking for feedback. I'd still prefer giving the feedback during the conference and directly to the concerned people but with a way of ensuring that this feedback will be tracked (whether it is used or not doesn't matter that much, what's important to me is that it is considered and that I know of the outcome).

All in all a very good week and in addition were had some nice treats: a welcome basket of eatable goodies and ... a one-year MSDN subscription! Now that's very nice and I'm pretty sure last year's participant must be jealous by now ;-). BTW, on this topic of presents, there were some real disapointments when MS announced that we would have a 120$ voucher to buy stuff at the MS company store ... and when we later learned that this voucher was only the right to spend up to 120$ at the store! I'd suggest to remove this next year as most around me (including me) have found this more negative than positive. Of course the announcement of the MSDN subscription the day after helped a lot overcome this negative feeling ;-). We're too spoiled for sure...

Once more, thanks Steve and Microsoft for this very nice week!

"During this Vincent Massol interview you'll receive more information on the status, philosophy and strenghts of Maven 2.0. "What were the shortcomings in Maven 1 and how do we now write maven 2 plugins ?" are just a few questions Dion Almaer asked. Other topics discussed are Continuum, Cargo and Agile outsourcing and offshoring... check it out."

The video is available here.

It appears that Microsoft is serious about Java and Open Source after all. They contacted me for my work on Maven. As a matter of fact, their own MSBuild system is not working as well as they would have liked and they are interested in reusing Maven as a build scaffolding and develop numerous Java and .Net plugins on top of it and integrate it all in the next version of Visual Studio (which will sport lots of new features for Java development too). This is all part of their new 2007 plans for implementing Software Factories. I can't tell you much more at this stage.

The best news is that I won't be working alone on this as they are setting up a full scale Java Team. Yeah that rocks! Some friends from our Open source group in Paris have also been recruited (check Ludovic's , Jeremi's , Didier's and Francois's blogs).

I'll be blogging more about it, stay tuned...

Update (3rd April 2006): Of course this was an April's fool. However there's some truth in it. I'm indeed going to Redmond the week of the 10th of April. Microsoft is inviting 70 people using competing technologies from all over the world. The idea seems to be twofolds: Microsoft will present what they are working on and we are expected to critique/provide feedback on what they're doing compared to what we are doing using our technologies. Sounds pretty fun. It seems this is something similar to the event that Matt Raible attended (more here, here and here). I'll blog more on this when I know more.

• Moderated wikis. Right now there are only two choices for a wiki: either they are open and anyone can edit a page or they are closed wikis and you need to register and get the rights to make modifications. For example most spaces on the Codehaus wiki are closed. They were initially open but vandalism was too high and we had to close them. This is hampering documentation contributions. A moderated wiki would alleviate this: when the page is saved, an email would be sent to a list of moderators for the space for approval of rejection (either by responding to a certain email address as for mailing list moderation or by clicking on a link in the email). Ideally, clicking on the validation link in the email would open the page in a browser with the modifications highlighted so that the moderator could make some changes before clicking on the save button.
• Anonymous edits. Although this feature already exists, I'd like wikis to add 2 fields when anonymously editing a topic: a user name and an email address. The idea is make it even easier to contrinute to a wiki. If the wiki is moderated as explained above, moderators would receive an email. The idea of the username and email is to allow the moderator/community to discuss with the contributor if need be and to give him credits. These 2 fields would obviously be optional and there should be a text on the page explaining that the email will not get displayed on the wiki and that filling the fields will allow credits/acknowledgment to be given.
• Diff notifications. Most wikis allow some form of space watch but the wikis I have used still do not offer the possibility to send notifications in a text diff format (wiki markup diff is good enough). For a development wiki, the idea is to send diff notifications to the development mailing list so that all developers are aware of wiki page modifications.
• Daily notifications. This is also supported in some wikis but what I would like is the ability to watch a single space and to aggregate changes in that space (using the diff notification format mentioned above). Please note that Confluence does not support this as it requires you to modify all other spaces permissions so that the user doing the watch has no view rights on the other spaces, which is not usable for example on wikis such as the one on Codehaus which have hundreds of spaces.
• In place comments. The idea is again to lower contribution by allowing wiki users to highlight a portion of text in their browser and to associate a comment with it (like a post-it). There would be an option to turn on/off these comments. It's easier for a user to highlight a line and put a comment like "I don't understand this sentence" or fix a typo rather than have to use current the type of comments at the bottom of a page. Note that this is similar to how word processors such as Word allow adding comments to a document.
• Patch handling. I'd like the ability to make modifications to a page and then instead of saving, have the ability to click on a "Generate patch" button which would generate a text file in wiki markup diff format. Then there would need to be a "Apply patch" action that can be done on a page. This would allow using wikis for project development web sites and allow contributors to provide documentation patches along with code patches. This is currently a big pain when using a wiki as a project development web site.

I have quite a few other suggestions for improvements but I feel those are the major ones when it comes to using a wiki as a project development wiki. Let's hope wiki vendors are listening... :-). Are these also on your wish list?

I guess it depends on the definiton of "successful". My definition of a successful OSS project is:

1. A project that has a big user community and mindshare
2. A project that has a big development community (i.e. committers and contributors). That is the development mailing list should be quite active with lots of contributions, improvements, ideas being discussed. All those submitted by a large number of different persons
3. A project that lives on if the initial or main contributor leaves the project

If we go by point 1, I think that the main 2 projects I have started (Cactus and Cargo) are not doing too bad. Cargo is still new but it seems to be on track and its user base is growing very quickly.

However on points 2 and 3, I'm not so sure I'm doing well. Mind you, Cargo has quite a lot of committers (growing every week!) and they're all doing a great job. I'm just thinking about the level beyond (look at the activity on the Maven project, Spring or other projects as an example of what I mean). Cactus is a bit different as it's now a mature project, so let's focus on Cargo. Of course it could be that these project domains are narrow and thus do not interest lots of developers. This is probably true but I don't think this is the only issue. I have the feeling that some of the reasons could be:

• Unconsciously I may be "driving" the project too much. For example if someone proposes something, I'm probably going to argue with this person in order that it fits Cargo's quality crtieria and direction. It's possible that by doing this, I'm cutting some creativity from this person and thus even if he implements the thing, maybe he'll feel it has not completely come from him and won't identify with it enough to maintain and improve it in the future. Or maybe by arguing I'm just making the life of that person harder and as this person is doing this in the little free time he has, he may not pursue it...
• Maybe I'm answering emails asked on the list too fast, thus preventing any other contributor to answer. This is setting the tone and maybe as a consequence people are expecting me to answer all emails. Everyone "knows" that I'll answer them anyway...
• The same could be said for applying patches and implementing things. I think this is less true though as I also have a day job and there are lots of JIRA issues accumulating in the Cargo project for example, so there's room for takers
• Documentation of Cargo seems good at first glance (although I know there are lots of holes). I wonder if, as a consequence Cargo users who look at the web site think that Cargo is stable, mature, etc and thus may feel less inclined to participate. Who wants to participate to a project that is mostly "done"! (Note: For those reading this and interested in Cargo, it is far from true and there are tons of things to design and implement!)

On the other hand, I feel that not doing any one of these points will hamper Cargo's user adoption...

Or maybe I'm completely wrong on all points above and this is just me fantasizing!

I'm really interested to know what you think and if this is something other OSS committers/contributors have noticed too. I've given the example of Cargo but really this is a general discussion on how to best manage an OSS project.

• When you're developing a framework, i.e. a piece of software meant to be used at an API level by other developers. In that case, breaking binary compatibility is not something to do lightly.
• When working in a large team it's common to define "interface" projects that represent the contracts to be followed by the different teams. In that case breaking the binary compatibility in an "interface" project is something that has to be planned and organized.

Enforcing binary compatibility in the build

The automated build is a nice place to enforce binary compatibility as the build is something executed by the indiviudal developers before checking-in and it's also executed by the continuous integration build. Thus any binary incompatibility can be quickly discovered. Or course this doesn't replace tests which can also help discover breakages. However the problem is that with all the nice refactoring IDEs we have now, it's easy to refactor the tests at the same time as the code and thus introducing a binary incompatibility is not always noticed.

A good strategy to discover an incompatibility is to compare the current code with the latest released code. This is what Clirr is doing. Clirr currently sports an Ant and Maven1 integration. The good news is that there's a Maven2 plugin in the work (more on that when it's released). However using a tool is only good if there's a strategy behind it.

Strategy for using Clirr

Here is what I believe can be done to automate binary compatibility checks in the build:

• Start by organizing your packages so that you clearly demarcate the user-public API from the SPI from the internal implementations. You'll probably want to fail the build only on the user-public API (and possibly on the SPI too but that's probably a lower severity).
• Use Clirr to make your build fail upon violation on the user-public API.
• After discussing with the team and possibly with users, decide whether you wish to allow the binary incompatibility. Always consider going for a deprecation cycle. If you choose to allow the incompatibility, register it in an exception file that you pass to Clirr so that it builds without choking on those errors (Note: I believe Clirr needs to be improved to better support exceptions not only at the file level but at the violation level).
• When the release time comes, you'll have a nice file listing all the binary incompatibilities. Include it in the release notes so that your users know what to expect and even better, for each incompatibility add a description that explains how to modify the user code to use the new version of the API.

Note: On the Cargo project we've tried to do this, even though there's still room for lots of improvement. Actually our main issue on Cargo is not detecting binary incompatibilites but rather deciding to release a 1.0 version which would mean that from then forward we would aways look for a deprecation solution rather than break binary compatibility. We've always pushed back this 1.0 release because our API has been changing quite frequently but we're now nearing a 1.0 version. When that comes we'll turn Clirr on to fail the build upon breakage. I'll let you know how it goes...

The way I work goes like this: I start writing about a topic. If it's taking too long to explain it, I consider that something is wrong about the code. I modify the source code so that the document I'm writing has the minimal required size to explain the topic.

The good thing with a book is that what you're explaining has to be simple and not convulted which leads to this nice effect of improving usability of your code. I get a bit of the same result when I write project documentation but not to the same level. This is probably simply because writing a book is a more involved process, you dedicate more time to it and thus you want it to be as perfect as possible (and thus as readable as possible).

I guess nothing here is new. This is all about having a user of your code. Tests are "users" of your code and thus leads to better design. I guess documentation can also be a "user" of the code and thus help improving it.

If you're writing some framework/tool, consider writing a book for it and if you're diligent in your writing your code will end up being better! As an added benefit your users will love you... :-)

Note: Adding a new container is very easy and you only have to implement a small interface and your container will be made available automatically through all the existing end user APIs - Make sure to contact us if you're interested in adding a new container support.

I'd like to quickly demonstrate how to use the new Maven 2 plugin on 2 use cases (more samples can be found here):

• Use case 1: Deploying a WAR to Tomcat 5.x and starting the container
• Use case 2: In-place webapp deployment with Jetty 4.x

Deploying a WAR and starting Tomcat 5.x

Create a Maven 2 project and put the following configuration in your pom.xml file:

To generate your WAR, start Tomcat and deploy the WAR in it, simply type: mvn package cargo:start . That's all! Do you want to do the same in, say, Orion 2.0.5? Simply change tomcat5x with orion2x and the home element to point to where you have installed Orion 2.0.5. You don't have Orion on the machine running the build? No issue, simply replace <home> with:

Orion 2.0.5 will then be automatically downloaded and installed the first time you run your build.

Inplace webapp development with Jetty

Let's imagine you're using the same project as above but this time you'd like to start Jetty and make it point to your webapp directory (i.e. src/webapp ) so that whenever you make a change to your webapp's sources Jetty automatically picks it up and serves it. See Brett's nice blog post on this. Simply modify your pom.xml as follows:

Then open a shell prompt and type mvn cargo:start . Jetty will start and monitor your webapp's dir for any change (Note that the same can be achieved using other containers too).

If you're interesting in learning more, check the documentation and join us on the Cargo mailing lists.

I was lucky to be presenting in Room 1 (the big one) and it was well packed. Here are the questions I asked the audience last year:

"The room was packed (I'd say around 400 to 600 people). Before starting with my session I've asked how many people are already using Maven and I've counted about 20 (but at that time the room was only half-packed), so I'd say it was about 3-5%. My second was "How many are planning to use Maven" and I got a resounding 3/4th of the people raising their hand. That shows that Maven is still in it's early adoption phase and that it has some great potential."

This year when I asked the audience who is using Maven 1, I got about 30%-35% hands raised. Wow! This is a huge boost from the 3%-5% of last year! There were also about 5% people already using Maven 2 so that's great too. It seems Maven's adoption is well on its way

Thanks everyone for the warm welcome during the presentation.

Enjoy the slides while waiting for Javapolis to broadcast the videos.

read (photo);
photoContainsHuman = callMechanicalTurk(photo);
if (photoContainsHuman == TRUE) {
   acceptPhoto;
}
else {
   rejectPhoto;
}

This is really like the Matrix except that the humans get paid a little bit of money (but in the end that's close to getting fed) and it's other humans that controlling the programs... until we have web services using other web services using the Mechanical Turk. Then who's controlling who is going to be hard to decide :-)

Source: Google Blogoscoped.

1. load a database schema in the instance
2. load data in the instance
3. start/stop a database instance
4. ability to create an instance from scratch

The ideal situation would be to find an existing Java framework that would already perform all or some of those steps. Then I could easily create a Maven 2 plugin wrapping it. So far I haven't been able to find such a tool. If you know any please suggest them!

Here's what I have found so far below. Please note that I have probably made mistakes while filling this table and I'd be happy to be corrected...

	Load schema	Load data	Start/stop instance	Create instance	Comments
DBunit
DDLUtils					I think DDLUtils is the old commons-sql project.
Derby ij					ij 10.1.1.0 requires the db2jcc.jar which is not on Ibiblio. I need to check the license to see if it could be uploaded.

Again, let me know if you know some tools that are not listed here.

If no such tool exist, an idea I have would be to add support for databases in Cargo. Indeed Cargo is meant for manipulating any kind of containers. It happens that the first type of container we've implemented are J2EE containers but it should work for any other type and the interfaces should remain the same.

WDYT?

This is a sad moment. Our friend and colleague Nicolas Chalumeau has departed this world on the 4th of October at the age of 27.

The Apache Software Foundation would like to express its condolences to Nicolas' family and friends.

Nicolas had been developing open source software at Apache for several years. He was participating in several projects including Jakarta Cactus and Apache Maven. But even more importantly than helping those projects, what made Nicolas stand apart was his kindness and his generosity. He was always open to discussion and was constantly helping others. On the Cactus project he was regularly the first to answer questions from newcomers trying to use Cactus.

Recently he had taken up the important task of helping migrate Cactus to use Cargo and he had been working diligently towards that task. Actually Nicolas had been doing such a great job on the Cactus project that the Cactus committers had unanimously voted him in to become a committer. Nicolas was voted an Apache committer on the 1st of August 2005. I had several discussions with Nicolas about this new role and he was extremely happy to be part of the Apache Foundation.

Nicolas, we'll miss you, but the work you have done for the community will be remembered forever and will live on. All our thoughts and good wishes are with you and your loved ones.

-Vincent on behalf of the Apache community

(we have also created a special farewell page for Nicolas on the Cactus website).

The Google Summer of Code is now over. I've had the pleasure of being a mentor for Codehaus. More specifically I've mentored the following projects:

• JBoss 3.x and 4.x support in Cargo. This project was successfully implemented by Nyoman Winardi (a.k.a. Win). Win started sending patches and over the course of the programme Win became a committer proper. The full support of JBoss 3.x and 4.x will be available in the next release of Cargo (version 0.7).
• JSR-88 support for Cargo. This project was successfully implemented by Lev Olkhovich who also became a Cargo committer. Lev implemented deployment of J2EE archives using JSR-88. In the process he started a conversation to refactor Cargo to support the notion of remote containers. There's still some refactoring going on and we need to add some more tests but we should be able to have support for remote containers and JSR-88 in Cargo 0.7.
• Refactor Cactus to use Cargo. One more project revolving around Cargo! This one was implemented by Xuan Thang Nguyen. Xuan sent several patches and thanks to Felipe we've applied some of those. However, I have to admit I have personally not been available enough to fully help Xuan apply his patches. We still have some patches in JIRA that haven't been applied yet. Actually the plan was to have Nicolas Chalumeau to help Xuan and apply Xuan's patches. Nicolas has been working and helping on Cactus for a long time now and was voted a committer on Cactus at the beginning of the SOC programme. However, the Apache Software Foundation (ASF) is extremely slow when it comes to adding a committer to a project (it can take more than 2 months) and we've not been able to give right access to Nicolas. Thus he's not been able to apply his own patches nor Xuan's... This is really an issue that the ASF has to solve quickly lest it'll see people leaving to create their project somewhere else.
• Faqbot project. There were 2 students on this project: Jie Tang and Harsh Puri. Harsh has had to resign from the programme because of the tragic flooding that happened in the region of Mumbai. Jie has continued alone and has done some good work. Unfortunately he's not been able to fully complete the project (which was probably the most ambitious of all the SOC projects I've mentored). The hardest part was probably starting a project from scratch. Everything had to be done. Hopefully Jie and others will continue the project and make it release-ready. I was very excited by this project and I still am.
• Real-time collaboration editing (Oxyd). This one was implemented by Jeremi Joslin of XWiki fame. Even though it was a project started from scratch Jeremi was able to complete it and have a first usable release ready. Well done Jeremi!

Learnings

• Open source is about collaboration with others. I don't think the SOC emphasis was enough on this point. For example it was "fordbidden" for students to work together and the main focus was to produce a working piece of software.
• Open source is not bound by time. People do it in their free time (at least most people) and as such they can't be expected to be bothered by strong release pressure. The SOC students had to work on a given date which caused some friction as the students were not always aligned with the project's timeline. Let me give you one example; It happened that some student needed to do a refactoring to the existing code to progress. This needs to be reviewed and possibly voted by project committer. There could easily be a delay of 1 week before we get everyone's agreement/ideas, etc. In the meantime the student is under pressure to quickly progress.
• I have taken too many students. I felt I did not do the best possible job when it came to mentoring them. Some were not autonomous enough and would have required more mentoring. I'll take fewer students next time. The hardest is really to mentor students on a new open source project started from scratch. As I'm already involved with several open source project, I did not have enough bandwidth to help on all aspects required to set up a new project.
• Several students had not enough time to participate. Some were still passing exams, others had some other summer job. This, combined with the deadline and the nature of open source did not mix well together. I think students should have an open source project to complete on a much longer timescale (possibly with milestones to monitor progress). This would also help in having them really integrated into an open source team. In addition it'll show their real commitment over time which is really what "professional" open source is about. There's nothing worse that someone who contributes big portion of code and then leaves some time after. Then, all the bug fixing and maintenance falls on the shoulders of the committers who were not the ones with the itch in the first place...

Parting words

The SOC was good. It has boosted the open source community quite remarkably (even though it has also probably put some strain on it...). Out of all the students I've mentored I think 2 or 3 of them (out of 6 initially) will continue to work on the open source project they've participated to. That's a 30%-50% ratio and I'm very happy about it. Thanks Google for making this happen!

I have presented From Maven 1 to Maven 2 which went well. There were about 60-70 people in the room, all Maven 1 users (to be expected for such a talk) and a few (about 5) Maven 2 users.

Update 2005-09-28: Kito has blogged about JavaZone 2005 and has uploaded some nice pictures.

Here are my findings after one day of using CDS. Please note that this is definitely not long enough to have a definitive opinion on the topic but I thought I'd still share what I've learnt today.

General opinion

CDS is a very good desktop search. I was very impressed. It seemed perfect at first and then slowly I started finding some little flaws compared to YDS. Still it is extremely good. It has all the features you'll find in YDS and Google Desktop Search (GDS).

Pros of CDS vs YDS

• Integration with Windows taskbar
• Low resource for indexing. It is not slowing my laptop when indexing. That's very good!
• Immediate scanning of new resources. If you receive an email for example, it is immediatly available for searching. No need to wait for the next indexing.

Cons of CDS vs YDS

• No vertical layout for views (as there is in YDS). This means that you cannot fully the message being previewed
• No "All" categories search. You have to choose the category you wish to search (emails, files, contacts, etc)
• No as-you-type results
• No possibility to choose the columns to display (for exemple email folders or email size). There are only a few basic columns
• Slower to search and display items than YDS. It was very fast initially and it quickly became slow and very slow as indexed items increased
• XML preview is using IE engine on Windows and thus there are lots of XML files that don't display correctly

Some minor details:

• Delete key does not work to suppress an email
• Cannot select different emails (to suppress them for example)

Conclusion

If only it could have a better view layout and be faster to display results it would be perfect. Its killer features are really its CPU-friendly indexing for me and the immediate availability of new resources in searches.

I've just noticed that YDS has released verson 1.2beta yesterday and I'm installing it. For now, I'll still keep using YDS which is still my favorite. YMMV.

It addresses different types of concern: from culture to communications to technical infrastructure to development practices.

Now just before you rush to it, you should know that it's written in French. I'm really sorry about that. I would have much preferred to write it in English but Pivolis' first market is currently France.

I hope you'll enjoy it.

Note: On the Pivolis web site, don't click on the English flag. The English version of the site is not yet up to date and you won't find the White Paper there.

In preparation for the launch of Maven: A Developer's Notebook, Tim and I have created the Mavenbook.org web site. By using it you'll be able to track the progress of the book but more importantly it'll allow you to download the book's source code, get additional samples, read tips and tricks on using Maven, etc. All of this through a blog interface.

BTW, we're not web designers so we've used a skin found on OSWD. I really like this web site which has both free skins and premium ones.

Last, the icing on the cake, the web site has been done using XWiki. Yeah, you heard it right, there is a wiki under the hood, which allow us to easily edit any page online. Each news item is actually a wiki page to which we have associated an Article object. The home page has a simple macro that does a search for all pages with articles and matching some criteria. Cool, no?

Enjoy!

For example, imagine that you have some integration tests in your project's test directory and that you need, say, Tomcat 5.0.30 to run them. You'll need to start the Tomcat container before the test:test goal kicks in. Do this by writing a preGoal in your maven.xml :

You'll need to provide Cargo configuration data in your project.properties or build.properties file. For example a minimal configuration would be:

cargo.containers = tomcat
cargo.container.tomcat.containerKey = tomcat5x
cargo.container.tomcat.homeDir = C:/apps/tomcat-5.0.30
cargo.container.tomcat.config.hint = standalone
cargo.container.tomcat.config.dir = ${maven.build.dir}/tomcat
cargo.container.tomcat.config.standalone.servlet.port = 8180

Alternatively you can ask the plugin to automatically download and install Tomcat for you (it'll download it only once), by specifying:

cargo.containers = tomcat
cargo.container.tomcat.containerKey = tomcat5x

cargo.zipUrlInstaller.tomcatinstaller.installUrl = http://www.apache.org/dist/jakarta/tomcat-5/v5.0.30/bin/jakarta-tomcat-5.0.30.zip
cargo.zipUrlInstaller.tomcatinstaller.installDir = ${maven.build.dir}/installs
cargo.container.tomcat.zipUrlInstaller = tomcatinstaller

cargo.container.tomcat.config.hint = standalone
cargo.container.tomcat.config.dir = ${maven.build.dir}/tomcat
cargo.container.tomcat.config.standalone.servlet.port = 8180

As you may have noticed, in our example above we've reused the existing Maven Test plugin which looks for test sources in ${pom.build.unitTestSourceDirectory} . With this strategy you'll need to create a separate subproject for running your integration/functional tests in order not to interfere with pure unit tests that you may already have in ${pom.build.unitTestSourceDirectory} .

If there's a strong demand, we may consider adding a cargo:test goal in the future that you look for tests in, say, src/test/cargo by default (leaving src/test/java for unit tests).

Please note that there's also an alternative which is to start the Container directly from your unit tests.

If you find Cargo interesting, please come and help us on the Cargo mailing lists. There are lots of different ways you can help: trying Cargo on containers, implementing new containers (for example, JBoss, JOnas, WebSphere, etc), discussing new ideas, letting us know what new features you'd love to see, etc.

The major changes from Cargo 0.4 to 0.5 are:

• New Maven plugin
• Added support for hot deployments
• Added support for the jo! container

Detailed release notes are available on the download page.

Here's an example of how to use Cargo from Java:

Container container = new Resin3xContainer();
container.setHomeDir("c:/apps/resin-3.0.8");

Deployable war = container.getDeployableFactory().createWAR("path/to/simple.war");
container.getConfiguration().addDeployable(war);

container.start();
// Here you are assured the container is started.

container.stop();
// Here you are assured the container is stopped.

Here's an example using the provided Ant tasks:

And here's an example using the Maven plugin:

// To run it:
maven cargo:start

// To configure it, add the following in a Maven properties file:
cargo.containers = myTomcat
cargo.container.myTomcat.containerKey = tomcat5x
cargo.container.myTomcat.homeDir = c:/apps/jakarta-tomcat-5.0.30
cargo.container.myTomcat.config.hint = standalone
cargo.container.myTomcat.config.dir = ${maven.build.dir}/myTomcat/config
cargo.container.myTomcat.config.standalone.servlet.port = 8180

Enjoy!

Here's the kind of output Clirr gives (the full output is available here):

ERROR: 8001: org.codehaus.cargo.deployment.DefaultJarArchive: Class org.codehaus.cargo.deployment.DefaultJarArchive removed
INFO: 8000: org.codehaus.cargo.module.DefaultJarArchive: Class org.codehaus.cargo.module.DefaultJarArchive added
ERROR: 7002: org.codehaus.cargo.container.Container: Method 'public void addDeployable(org.codehaus.cargo.container.deployable.Deployable)' has been removed
INFO: 7011: org.codehaus.cargo.ant.ConfigurationElement: Method 'public void addConfiguredEar(org.codehaus.cargo.ant.EARElement)' has been added
INFO: 4000: org.codehaus.cargo.container.jetty.JettyStandaloneConfiguration: Added org.codehaus.cargo.container.configuration.StandaloneConfiguration to the set of implemented interfaces
ERROR: 7005: org.codehaus.cargo.container.Container: Parameter 1 of 'public void setConfiguration(org.codehaus.cargo.container.Configuration)' has changed its type to org.codehaus.cargo.container.configuration.Configuration
ERROR: 7006: org.codehaus.cargo.ant.ConfigurationElement: Return type of method 'public org.codehaus.cargo.container.Configuration createConfiguration(org.codehaus.cargo.container.Container)' has been changed to org.codehaus.cargo.container.configuration.Configuration
ERROR: 4001: org.codehaus.cargo.container.jetty.JettyStandaloneConfiguration: Removed org.codehaus.cargo.container.Configuration from the set of implemented interfaces
INFO: 7003: org.codehaus.cargo.container.spi.AbstractConfiguration: Method 'public void configure()' has been removed, but an inherited definition exists.
ERROR: 5001: org.codehaus.cargo.container.deployable.EAR: Removed org.codehaus.cargo.util.MonitoredObject from the list of superclasses
INFO: 5000: org.codehaus.cargo.container.deployable.EAR: Added org.codehaus.cargo.util.monitor.MonitoredObject to the list of superclasses
ERROR: 7012: org.codehaus.cargo.container.Container: Method 'public java.io.File getOutput()' has been added to an interface
INFO: 7010: org.codehaus.cargo.container.spi.AbstractContainer: Accessibility of method 'protected java.io.File getOutput()' has been increased from protected to public
INFO: 6000: org.codehaus.cargo.container.property.GeneralPropertySet: Added public field JVMARGS

Even though we're using JIRA with an Iteration-Driven Development strategy (IDD) it was still a very interesting exercise to verify that we had not missed any issue by running Clirr on the source code. In addition, it provides a more detailed view of what exactly has changed in term of API which our JIRA report does not provide.

The next step would be to use it to fail our build whenever someone introduces a public API break. It would be quite easy for us because we've cleanly separated non-public API from public APIs by using internal packages (see the Cactus API design rule to see what it means). Of course sometimes, you want to voluntariy add a breaking change. That's legitimate but it has to be controlled. The strategy would be to have the build fail and then if the change is voluntary to exclude it from Clirr.

Well done Lars!

I think this is really where the value of Ant is. All those years of implementing the base building block for a portable OS Java API have created a very useful Task set. I think every Java application that needs to do copying, deleting a directory, spawning a Java application, etc should use these tasks. There's no point in reinventing the wheel!

For example, you may think that deleting a directory is simple. But it's not so easy. Have a look at the Delete Ant task source code. You'll find portion of code like this one:

/**
 * Accommodate Windows bug encountered in both Sun and IBM JDKs.
 * Others possible. If the delete does not work, call System.gc(),
 * wait a little and try again.
 */
private boolean delete(File f) {
     if (!f.delete()) {
          if (Os.isFamily("windows")) {
               System.gc();
           }
          try {
               Thread.sleep(DELETE_RETRY_SLEEP_MILLIS);
           } catch (InterruptedException ex) {
               // Ignore Exception
           }
          if (!f.delete()) {
               if (deleteOnExit) {
                    int level = quiet ? Project.MSG_VERBOSE : Project.MSG_INFO;
                    log("Failed to delete " + f + ", calling deleteOnExit."
                        + " This attempts to delete the file when the ant jvm"
                        + " has exited and might not succeed."
                        , level);
                    f.deleteOnExit();
                    return true;
                }
               return false;
           }
      }
     return true;
}

Would you have thought about this? Probably not and you would have been right not to as this only happens in some rare occasions. But when one of your users reports it, it's going to be darn difficult to identify and fix. Personally I'd rather depend on a stable and well tested library rather than recode it myself.

The problem is that the Ant tasks are a bit too much linked to the execution engine (the XML scripting engine). For example reusing an Ant tasks requires you to create a Project object. This in turn drags loggers, the Ant classloader (in some cases) and possibly other objects. I know it's possible to use Ant from Java (I've been doing it for a long time now) but I'd love it be even easier to do so.

Instead of writing:

Project project = new Project();
Expander expander = project.createTask("unzip"); 
expander.setSrc(new File(zipfile)); 
expander.setDest(new File(destdir)); 
expander.execute();

I'd like to be able to write:

Expand expand = new Expand();
expand.setSrc(new File(zipfile));
expand.setDest(new File(destdir));
expand.setLogger(myLogger);
expand.execute();

I don't want to see the get/setLocation, get/setTaskName(), get/setDescription() and in general all methods from Task.java.

What I'd love to see is Ant moving in the direction of providing completely reusable Tasks that have 0% dependencies on the Ant engine. This means that loggers, classloaders would be passed to the Ant task by the program who uses it.

I'd like to see Ant provide 2 distributable jars: one containing the XML scripting engine only and one containing all the pure java beans Ant tasks that can be reused in any Java application.

I'd like to see Ant separate into 2 subprojects: one for the XML scripting engine (let's call it engine) and one for the Ant tasks (let's call it tasks). The reason for the 2 projects is to ensure there's no dependency in the direction tasks->engine.

I'd like to see Maven2 use those completely reusable Ant tasks instead of recreating them (this is a wish I'm addressing to both projects, not just Ant! :-)).

I'd like to see those Ant tasks being a JSR and incorporated in a future version of the JDK, thus providing a higher level API that the best classes from the JDK.

Is that where Ant is heading today?

Indeed, the barrier to participation is often due to the fact that a user who wants to participate will need to understand the whole design, how the different classes are entangled, what effect a change here will have on the rest of the project, etc. Thus, if we make the project more modular a contributor who wants to participate will only need to understand the design of a given 'module'.

A 'module' would need to have some good-to-have characteristics:

• Very loose coupling with other modules
• Clearly defined and *published* interfaces. There should be some documentation on the project's web site explaining them and a tutorial showing how to implement new modules (or swapping a module implementation by another one) for example.
• Separate builds so that it's easy to build only the module (this can be alleviated if the master build is easy to use (i.e. no property tweaking necessary, it just builds - As it's the case with good Maven builds... ;-))
• Separate documentation on the web site, so that the website itself is modular and the complexity of each module is hidden in that module's web site. Thus the top level web site would be quite simple only listing what the project does as a whole and listing the different modules

Interestingly one way to implement the 'very loose coupling with other modules' characteristic is by using a Service Architecture. This can be done for example by using the Dependency Injection pattern and/or using a lightweight container - PicoContainer, Spring, etc).

This is probably obvious stuff but I've just realized that it's not only good design practices but that it'll also help open source projects attract more contributions. Of course that leads to another topic which is when to accept contributions and how to maintain them in the long run but that would be another discussion...

So what is the need for a distributed build?

I can see several use cases:

• building on several JDKs
• building on different OS platforms
• building with different environment setups (for example building with different application servers, different browsers, different databases, etc) to validate that a product integrates well with various environment setups
• delegating the build load on several machines when the build starts to take too long (of course, the first solution should be to try to lighten your build as much as possible)

A proposed architecture

Disclaimer: this only ONE potential solution. There are lots of other solution probably even more valid than this one. Please feel free to add your ideas as comments to this post.

It could work as follows:

1. The central build machine (aka the build orchestrator) decides to start a build. The orchestrator can be one existing continuous integration tool like CruiseControl, DamageControl, etc. They can trigger a build on anything they want: time-based, change-based, manual, continuously, etc. The orchestrator sends a build request to the space. The request contains all the information about the requested build (e.g. JDK to run on, OS to run on, App.Sever/DB/etc to run on)
2. The space holds all requests. It chould be a good idea to provide a browser to see pending requests (preferably using a simple HTTP browser so that people who wish to contribute can see what type of builds are required). In any case it's important that the space be transactional (Note: I'm not sure about the word "transactional". What I mean is that a request cannot be read by several build agents at the same time)
3. Build agents listen on space build requests objects that match their capabilities. Using Jini/Javaspace would be nice here because (among other things) agents would be able to easily listen to requests with Jini attributes (OS, JDK, etc). Once they read a request they start a local build and publish the result to the space as a Result object
4. The build orchestrator listens to Results object, and generate result reports, aggregating all results. Build results could contain anything required: result of the build, logs, generated artifacts, etc. The orchestrator gets the data from the Result object and perform usual build operations (publishing, build result notification).

Of course there would be several details to sort out, like should we send 2 Requests object for each build need so that we can compare the results and only accept the result if they match, etc.

Conclusion

I think this type of distributed build could be especially interesting for open source projects in order to build an active community around a project. This would be yet another way in which people can contribute to an open source project: by lending some of their machine CPU to perform continuous integration builds of this project. This usually makes sense as open source projects may be low on hardware resources and lending some would help. Of course it also bring its challenge of security issues that would also need to be implemented...

Would you like such a distributed build system? I personally prefer this architecture over one where the orchestrator directly sends build requests to build agents as I find it more scalable and more flexible.

I've also written 2 Maven quizzes for JavaBlackBelt.com. If you want to see if your Maven knowledge is up to the par, go and take them! And let me know your comments. If you register on the site you can even propose new questions!

For IntelliJ IDEA

• Start by downloading the latest javasvn.jar from the TMate JavaSVN web site. You need version 0.8.0 or later (I've used 0.8.0). The reason is that there is a new property named javasvn.ssh2.key that has been added as a hack for getting the Subclipse plugin for Eclipse to work with svn+ssh... Drop the jar in [IDEAHOME]/plugins/svn4idea/lib , replacing the existing jar of the same name there.
• Modify the [IDEAHOME]/bin/idea.bat file to add the javasvn.ssh2.key system property: IF "%IDEA_JVM_ARGS%" == "" set IDEA_JVM_ARGS= [...] -Djavasvn.ssh2.key=/path/to/your/private/key .
• Make sure you use an openSSH-compatible private key. The Putty format is NOT supported by JSch. If you have a Putty private key, use Puttygen to export it as an OpenSSH key.
• Make you sure you specify a valid username in the IDEA subversion setting, leaving the password field blank.

For Eclipse

• Install Subclipse using the provided Eclipse update site
• Install the JavaSVN Subclipse Extension, also using the provided Eclipse update site
• Modify the way you start Eclipse as mentioned on the JavaSVN Subclipse Extension web page
• Go the SVN Repository Exploring perspective, right-click on your SVN Repository, click on "properties" and make sure you enter a valid username (leave the password blank).

Enjoy! That should please all the Codehaus hausmates... :-)

Enjoy! (I hope...;-))

The typical local builds that developers run on their machines work by building the subproject they're working on but also all the dependent subprojects it requires. Usually, as building all dependent subprojects takes a lot of time, the developer infrequently checks-out other project sources and build them on demand. His focus is on his subproject that he's making modifications to (and rightly so!). This strategy has the following drawbacks:

• Setting up the build on a new fresh machine is complex and takes time. Indeed you have to check out all the top level project sources and build all projecets one by one until you reach the subproject you're concerned with.
• It doesn't scale too well. Your local build starts taking tens of minutes which does not encourage running it that often. And if you do, you don't rebuild all the subprojects even though there are probably lots of changes that have been made by other coworkers. Thus, you're increasing the possibility of an integration break (breaking your other coworkers when they integrate your changes).
• When someone from another team inadverently breaks your project's build, you'll have to switch context (i.e. stop what you're doing) and help out to restore the master build. If this happens unfrequently, it's probably fine and even positive (as it increases team collaboration ;-)). However when it happens frequently (which is bound to happen as the team grows), you'll start suffering from it...

Because of all these problems, I have been using a different approach on my current project for the past 2 years. This was mostly motivated by the fact that the project is a big project (close to 100 developers) and we were hitting the issues mentioned above. I have called this strategy "Binary Dependencies Build". If you're interested this is an approach I have presented both at TSSS2004 and at Javapolis 2004.

Here is how it works (click on image for a larger picture):

Imagine that you have a "trading" subproject that depends on 2 other subprojects ("partners" and "referenceData"). The idea is that your local build will NOT build them from sources but instead will download their latest version that work from a remote artifact repository (a location where the result of the subproject build is located). In order to accelerate even further the build, the versions downloaded are stored locally. In our example, the latest "partners" jar is already available locally and is thus not downloaded but the "referenceData" one is not. It is downloaded and then stored locally. The "trading" subproject is built using these binary dependencies.

This is all fine but there is a burning question: How do I do continuous integration with such a system? Won't the binary dependencies be old versions when I get them? The solution to this is to have a continuous build server that continuously build subprojects and puts their artifacts in the remote repository. Note that there are put in the repository only if their build passes with no errors. This ensure that there are always fresh versions available and that they are as "good" as they can get.

Doing it with Maven

The good news is that this feature is built in Maven. Maven implements this support of artifact repositories (local and remote) and it supports the process of automatic download of artifacts not available in the local repository. Usually Maven will verify first in the local repository if the artifact's version exists and if so will use it. However, if an artifact's version contains the "SNAPSHOT" keyword, Maven will always check if there's is a more uptodate artifact in the remote repository. This allows implementing easily the strategy defined above.

Conclusion

We've been very happy with this solution so far. I think there are 2 key points in making this work:

• A good build that provides assurance that the binary artifacts are working. Indeed we've experienced that our subproject build was not always good enough to qualify how "good" was a jar artifact. This was usually caused by the non-existence of automated functional tests which meant that even though the build was passing the jar was not working when executed on the developer's machine. The solution is of course to include integration/functional tests in the build (at least the master CI build).
• A quick master build. It's important that it generates fresh jar artifacts as quickly as possible so that CI can happen as often as possible.

1. Build takes too long to execute (and thus it is executed less often)
• Fix the build by having more subprojects with binary dependencies and/or streamline the build to ensure that only important build steps are run. Optimize it (f.e. offer different goals/targets: one for a clean build and another one that does not perform a clean).
2. Local build not executed
• Build coaches to increase build awareness. Complementary idea: "unbreakable builds".
3. Public API breakage in dependent project without warning
• Better team communication. Set up a deprecation policy. Complementary idea: "unbreakable builds".
4. Not enough continuous commits (all packed up at end of iteration)
• Team meetings to explain more the importance of continuous integration. Complementary idea: "unbreakable builds". The idea is that if you keep your changes to yourself and accumulate them, whenever you'll want to commit them, the unbreakable build will likely reject your changes as they will break some other part of the code. Thus you'll need to spend several days to talk to other developers to not only fix your code but also fix theirs. Normally after doing this several times, you should understand that it is in your best interest to commit frequently.
5. No functional/integration automated tests (f.e. no local verification of ejb-jar deployments)
• Automated functional tests! Build a suite slowly over time, improving it at each iteration. And maintain it! Decide on a good data handling strategy (this is usually the main issue). Ensure that your data strategy keeps everyone in sync WRT DB data.
6. Commit problems (Forget to commit some files, Pb due to SCM tool - Starteam: new directory do not appear in Starteam view!)
• Coaching + change the SCM tool! Complementary idea: "unbreakable builds".
7. Devs “building” with IDE but forgetting to use the automated build
• Coaching. Complementary idea: "unbreakable builds".
8. Checkstyle errors failing the build
• Coaching. More team meeting to decide what checkstyle errors we want to fail the build or not. Get a strong team buy-in. Complementary idea: "unbreakable builds".
9. Failing unit tests
• It probably means that the unit tests are actually integration tests depending on database data. Ensure that unit tests are quick and fast and independent of the environment. Complementary idea: "unbreakable builds".
10. [Maven] project.xml not up to date and missing dependency
• SCM diff emails on check-ins (team by team) in order for everyone to have the knowledge of what's happening. Complementary idea: "unbreakable builds".
11. Database data modifications (voluntarily or involuntarily) leading to test breakage
• Better data strategy. See for example the databse testing strategy in build post.
12. Continuous build not cleaned between different runs
• Fix it. Perform a clean build from time to time.
13. Local SCM update not done before local build (in order to get the latest files)
• SCM diff emails on check-ins (team by team) in order for everyone to have the knowledge of what's happening. So you'll know better when to update our local workspace. Complementary idea: "unbreakable builds".
14. Environment differences in local build vs central build
• Work continuously towards making the developer's environment as close as possible as the integration environment. Complementary idea: "unbreakable builds". This allows executing the build on the server and thus it runs in the same environment as the continuous build.
15. No local deployments done before commits (f.e. no EJB deployments)
• Coaching (in order to ensure that developers do perform deployments on their machines before check-in) + add some checks in the build to automate the verification (they can be f.e. some hand-picked functional tests).
16. Checkstyle errors hidden in tons of warnings
• Fix it. Newest versions of Checkstyle allow filtering on severity.
17. Non-atomic commits and central build starting with in-flight commits
• Use a scheme a la CruiseControl (wait for some inactivity time on the SCM before trigging a build). Or change the SCM (for subversion for example). Note: We have tried to use CC with StarTeam but even though the infrastructure team increased CPU + RAM, StarTeam falls when it is polled by 3 or 4 CC builds in parallel... (Solution: Dump ST or ask Borland to come and tune the parameters). Complementary idea: "unbreakable builds". This forces "atomic" commits.
18. [Distributed development] rsync issues: sometimes jars are corrupted or lost
• Fix the rsync process (Note: this is now no longer happening I believe)
19. [Distributed development] VPN instability making it difficult to SCM-update
• Fixed mostly. However usage of Starteam is still extremely slow making hard to SCM-update from remote. Solutions: Use a less bandwidth/responsetime consuming SCM (f.e. Subversion), increase bandwidth (but the issue is mostly with response time which cannot be changed), or use a replication mechanism (I don't like this as I believe it introduces its own issues - I really much prefer everyone working directly on the same repository, especially as I know it works (I've done it in the past using CVS with a team of 30 developers and it was working fine).
20. Errors when executing the application
• This is because there are no automated functional tests. Automate them!

Out of my last two development projects, one had a strong sense of quality and excellence in general and continuous build failures were the exceptions (about 3-4 per week for a 30 developers team) and the other one was quite the opposite and everyone was surprised when the continuous build was passing (there were about 5 build breaks a day as an average for a 40 developers team). I'm sure this is also pretty common to other projects. Obviously the best is to build (pun intended) a build awareness in the team. However, you'll need strong evangelists for this to happen who may not always be available and other circumstances may make this difficult.

A thought struck me about a year back: what if we were able to prevent the continuous build from failing by design. There's a French saying that goes something like "it's better to cure than to heal". I think this is definitely a good idea to apply to continuous build failures. Why not make a continuous build system that cannot fail. At that time I thought it was a nice idea (I had meant to blog about it but I forgot) but I could not see very well how it could work. Now a year after, I really think it's a nice idea and I'd like to explore it.

The architecture

A potential basic architecture is shown in figure 1 (click to get a larger picture).

The general principle is to catch the commit data before they get committed to the SCM, to perform a build and to perform the actual commit only if the build is successful. Here are the detailed steps:

1. The developer performs a commit using his favorite SCM client tool. Note that it is best if the tool is able to perform the commit asynchronously so that the developer can continue working on something else.
2. The committed data are intercepted using a pre-commit hook script (all modern SCM support this). This script is in charge of doing 2 things:
   • Finding out the list of projects to be built. Indeed, say that the commit contains 5 files belonging to 2 different projects. We need to rebuild these 2 projects. The algorithm for finding out the projects to which belong the changes sources can be as simple as a mapping between the file paths (which contains the project name) and the project name.
   • Creating a build job and pushing it on a queue. The reason for the queue is that building all the projects on the machine that hosts the SCM is not going to be scalable. We want the SCM to be as responsive as before. Hence the queue.
3. We need build machines to perform the actual build. They could be dedicated build machines that build continuously the build jobs. There could also be developer workstation. The concept is to have one or several build kicker applications installed on those machines. The "continuous build kicker" will continuously get a job from the build job queue and build it, whereas the "idle build kicker" will only pick a job to build when the machine is idle (hey, look around you and see how many machines are unused because the people are either on holiday, sick, in a meeting, etc. That's a lot of power).
4. The build kickers start by updating their workspace to have the latest files for the projects associated with the changes files. Then they try to "merge" the changes files in their workspace (note: this may be the tricky part to implement unless the SCM offers a way in the pre-commit hook to get the full file - I need to explore this). If they cannot succeed they stop with an error message that flows back to the user. This can happen if someone else has been working on the same source and their change has made it to the SCM before ours has. If the merge succeeds, the build kicker starts the build. The build hasa to be relatively quick so you should not build all the projects. I suggest building the modified projects and the ones that directly depend on them so that an API break can be detected (more on that below)
5. When the build is finished (or if an error occurs), the build kicker sends the result back to the pre-commit hook (using a RPC mechanism for example).
6. If the result if positive, the pre-commit script either performs the real commit to the SCM
7. The resulting message is returned to the user. In case of error the user would see for example the build console log

Advantages

Here are the following advantages of such a system:

• Does not break other developers upon a build failure. All developers can work uninterrupted even though they can still work on HEAD in a continuous integration fashion
• Lowers the effort required to get a CI system working thus it helps teams adopt CI
• Prevents breakage of APIs. Indeed in step 4 above, we've mentioned that a good strategy is for the build to build not only the projects that have changes but also all projects that directly uses those projects (one level). This will allow detecting unwanted API breakages.
• Increase self-confidence when committing which (I hope) will make it easier to get developers to commit continuously
• Allows continuing working on one's own machine (instead of having to wait for the current build to free the CPU which is being used at 100%!). You know get your own PBS (Personal Build Server)
• Forces atomic commits!

Questions/Issues

I'm sure you're now burning with tons of remarks/questions showing why it wouldn't work :-) Here's what I've currently thought about. If you have any opinion or other questions, I'd love to hear them.

Q: What happens if someone else also commits a change to the same file?

It works in the same way as usual. The build kicker will try to "merge" the changes after having done a workspace update and if it cannot, the user will get an error explaining that the merge failed. The user will then need to perform an update on his local machine and resolve the conflict.

Q: Imagine I perform a commit and I start working on a new feature. Then my commit is rejected because of a failure. How do I fix this without loosing my current changes?

Answer 1: This is actually relatively similar to what you're currently doing. Imagine you're committing something. Then you start working on something new and the continuous build tells you 2 hours later that your change has broken something. The difference is that your changes have been committed so you can easily create a new workspace and fix it there. We could do the same here by having the pre-commit hook actually make your changes available through a URL (sent in the commit answer) as a patch so that it is easy for you to apply it to a fresh new checkout.

Answer 2: You wait till the build is finished on the server. You can perform other activities like documenting, reading, thinking, designing, writing new classes, new tests, etc. Basically you work on stuff that do not conflict with the past changes. Actually this is probably what you're currently doing when your build is running as it is eating all your CPU...

Q: Doesn't it take too long to build?

You need to ensure your build is taking as little time as possible. I think 5-10 minutes should be ok. The best way to achieve this is probably to use binary dependencies instead of rebuilding dependent projects (a la Maven), except maybe direct dependencies. You'll still need a continuous build running continuously to produce fresh binary dependencies. I guess it's also best to use an SCM client that can do asynchronous commits in order to let you continue working while the commit is in progress.

Q: What if I want to modify an API but I wish that each projects modifies its own files?

Several options:

• You could go through a deprecation cycle.
• You could be doing the refactoring on one machine only (not always possible)
• You could also plan it. Anyway an API breakage has to be planned with communications. Thus you could say: on that day, at such hour we're going to be committing this break and we have 1 day to fix all our dependent projects. When this happens you can turn off this "unbreakable build" feature for the day.

The interesting point here is that you *want* the API breakage to be detected as the default instead of the opposite.

Conclusion

It seems to me this would be particularly useful on big projects with lots of developers. It should also be useful to introduce continuous integration on an existing project as it lowers the discipline required by everyone. Obviously this is just an idea that I haven't tested yet. I'm very keen to see this in action. If any of you has any experience please share it. I'm planning to spend some time trying to implement it. If you're interested to help out, let me know too.

There were the usual suspects (Mike Cannon-Brookes, Rod Johnson, Cedric Beust, Gavin King, etc) but as it was happening close to France, it was cool to see that almost all my OSSGTP fellows were also there (Ludovic Dubost, Henry Story, Benjamin Mestrallet, Francois Le Droff, Didier Girard). In addition it was good to see Jerome Lacoste and Philippe Kernevez there.

I've had the pleasure of giving 2 sessions:

• Maven: A session explaining why you would use Maven and trying to show Maven from all its different angles. I hope I have been successful in sharing my enthousiasm for Maven. The room was packed (I'd say around 400 to 600 people). Before starting with my session I've asked how many people are already using Maven and I've counted about 20 (but at that time the room was only half-packed), so I'd say it was about 3-5%. My second was "How many are planning to use Maven" and I got a resounding 3/4th of the people raising their hand. That shows that Maven is still in it's early adoption phase and that it has some great potential.
• AOSD: Agile Offshore: The last day was for the business track presentations and I gave one which is my return of experience of 3 years of doing offshore development using an agile methodology.

It seems the tracks were all recorded and you should be able to see them live very soon (I'm really excited to see that for myself as I've missed a few sessions during the first days of the conference and looking at oneself in Video is a good way to improve one's presentation skills... ;-)).

See you next year at Javapolis 2005!

• Issue smell 1: Too many unscheduled issues. This means that new issues are not assigned to iterations, i.e. that they are not planned to be fixed.
• Issue smell 2: Open issue from past iterations. Any issue that is left from a previous iteration has to be rescheduled so that everyone knows when it is planned to be fixed. If some portion of the issue has been done, I've found that it is usually best to split the task into 2, so that the work done in the iteration it was scheduled is clearly shown in the release notes for that iteration and the unfinished part can be scheduled in a future iteration.
• Issue smell 3: No iterations in changelog view. This means that past iterations that are finished have not been JIRA-released. The good thing about releasing an iteration is that it forces to solve the unfinished issues (see Issue smell 2). In addition it allows cleaning the roadmap view that becomes less cluttered by all past issues and which gives a clear view of what's left to be done. Last it provides an important feeling of achievement.
• Issue smell 4: Issue types in issue description. I have often noticed that some JIRA projects were using some description conventions for some issue types. For example, using XXX - Code review for a code review issue on the XXX feature. In that case, a real JIRA issue type should be created. The reason is that by defining a proper JIRA issue type, it is now possible to perform operations on this new issue type: it will appear properly in the release notes under its own category, it can be searched for, etc.
• Issue smell 5: Issue status are not in sync with the reality. This is often a big problem (especially with distributed teams) as people usually rely on JIRA to provide an exact view of the progress. If issues are found not in sync, there's a tendency to not "trust" JIRA anymore, which in turn leads to less using it and loosing visibility. One good strategy is to do Issue Driven Development (IDD). It goes like this: When a task is done and just before the code is checked in, ensure that the corresponding JIRA issue is marked as Resolved/Closed. If there's no issue, create one (unless the modification is a really minor one that the user should really not be concerned with). Then check in the code mentioning the issue number in the checkin comment (that allows for example using the JIRA CVS/Subversion plugins). Note: If you're using CVS/Subversion you could write a quick pre-commit hook that verifies that each comment has a reference to a JIRA issue.
• Issue smell 6: Lots of resolved (but not closed) issues. Most projects I have seen do not use a Resolved state. However, people often mark the issue as resolved but not closed and the issue stays in this state for ages without anyone doing anything about it. So either remember to directly close issues or if you're using JIRA 3 create a custom workflow that do not have a Resolved state (if you're not using the resolved state of course!).

Let me know if you have found other important issue-smells when using JIRA!

Automated tests are good. Automated Functional tests are even better as they are the proof that your application is working. In addition, with automated functional tests you can also automate your delivery process. However, writing automated functional tests is hard. The main reason it is hard is because you need to control your execution environment (database, application server, etc).

Cargo is a framework that you can use to automatically install, configure and execute J2EE containers. Thus it allows you to control your execution environment (for the J2EE container at least) and permits completely automated functional tests for J2EE applications.

Example

Let's walk through an example. Imagine we wish to start up Tomcat 4.1.31 before such test runs. Here's what we could write:

public class MyTest extends TestCase
{
     private Container container;
 
     protected void setUp()
     {
          // (1) Optional step to install the container from a URL pointing to its distribution
          Installer installer = new ZipURLInstaller(
              "http://www.apache.org/dist/jakarta/tomcat-4/v4.1.31/bin/jakarta-tomcat-4.1.31.zip");
          installer.install();
  
          // (2) Create the Cargo Container instance wrapping our physical container
          container = new Tomcat4xContainer();
          container.setHomeDir(installer.getHomeDir());
      }
 
     public void testSomething()
     {
          // (3) Statically deploy some WAR
          Deployable war = container.getDeployableFactory().createWAR("src/testinput/my.war");
          container.addDeployable(war);
  
          // (4) Start the container
          container.start();
  
          // (5) Perform any test you wish here
          [...]
      }
 
     protected void tearDown()
     {
          // (6) Stop the container
          container.stop();
      }
}

Step 1 is optional. You can also rely on the container being already installed on the test machine if you wish. However, it's nice to completely automated the testing and assume nothing (or very little - We still need an OS and a JDK on the machine). In this example we're fetching the Tomcat 4.1.31 installation from the web. We could fetch it from our intranet or from a location on the machine or from our SCM.

In Step 2, we have not told Cargo what container Configuration to use. Thus Cargo will use a default Configuration and it will configure it so that your container will execute in a temporary directory that it will create in your OS system tmp dir. If you wish to control this you can use:

Configuration configuration = new CatalinaStandaloneConfiguration(container, "target/tomcat4x");
configuration.setProperty(ServletPropertySet.PORT, "8080");
container.setConfiguration(configuration);

in step 3, we create a Cargo wrapper around a physical WAR and we add it to our container so that it is deployed when the container starts.

We then start the container (step 4), perform any testing we wish (step 5) and ensure the container is always stopped at the end of our test (step 6).

If we wish to start and stop the container only once during our whole test suite we can use a standard JUnit TestSetup.

Conclusion

This is just a short introduction to Cargo to demonstrate how easy it is to start/stop a container. The API is of course richer. Also, we're showing here how to use Cargo for functional testing of J2EE application but Cargo is also meant to be used by any application that requires a container to be up and running. It could also be used by IDE plugin writers, etc.

For more information on Cargo, please see the Cargo website and join us on the Cargo mailing lists. You'll be warmyl welcomed! :-)

At that point, either the library is open sourced and you can rush to download the source, modify the code to add some System.out.println and spend 3-4 hours to find out how to rebuild the project.... or it's not open source and then there's much that you can do except trying to find out the reason with your sheer brain power!

How good would it be if there was an application (let's call it a Logifier) at which you could throw a jar and it would return a new aspectified jar on which it would have weaved some Logging aspect that you could configure!

This would allow us to realize the full power of aspects: an external Java application that was not built with logging can now be converted to log things for us...

So who wants to be the first to build such a handy application? :-) Does it already exist?

Update 7/11/04: I've just remembered reading about AntFlow on TSS. That would be an excellent way of implement this. Imagine a hot folder called "logifier" and any jar you drop in there is automatically logified using an AspectJ/AspectWerkz/etc Ant task and a common logging aspect such as this one! Now that would be cool. It could be a good coding exercise for the next OSSGTP.]]> Think tank vmassol 2004-11-05T15:50:50+00:00 http://blogs.codehaus.org/people/vmassol/archives/000874_ossgtp_4th_meeting.html OSSGTP stands for Open Source Software Get-Together Paris. It's a montly meeting set up in Paris gathering Open Source actors (mostly Java OSS ATM). We've had our 4th meeting on the 21st of October 2004.

This time we had several newcomers: Henry Story, Patrick Chanezon and Luis Arias. Patrick did a presentation on Rome, his pet project and Henry tried to explain to us how to combine ATOM and FOAF and what ATOM/OWL was... Not a simple task is you're not versed in ontologies... ;-)

Here are Henry and Luis:

Here's Patrick:

And some of us trying to figure out what OWL, FOAF and ontologies are:

Then, as usual we went for dinner together and had a very nice evening disucssing a bit about everything...

A few other pictures are available here.

Next time we'll try some coding or review session as suggested on our Wiki. If you're in Paris, feel free to join us.

I've just tried Omea Pro (build 353) and I've got to say it's very promising! It's hard to explain what Omea is... I think it can be viewed as two things:

1. A search tool that aggregates all data from your computer (all files types including PDF, Word, Excel but excluding PPT, Outlook emails, ICQ/Miranda conversations, Outlook Tasks, Outlook contacts, RSS/Atom feeds, Newsgroups, etc)
2. A productivity tool that you can use instead of all your different tools for managing all your incoming data (mails, files, feeds, newsgroups)

After using it for 2 days, here are the pros I have found:

• The searching feature is excellent. I find it much better than Google Desktop or Lookout in term of relevance, breadth of search and organizing the results

Here are the thing to improve I have noticed (please remember that it is beta software):

• It's resource hungry: after a few hours of using it, it easily reaches 300MB and more. It's also a little bit slow.
• It has not reached a level where it is at least as good as the tools it gets its data sources from. For example, it's not as good as Outlook, it's not as good as a dedicated RSS feed reader, etc. I believe it will never be able to be as good as those specialized tools. JetBrains has recognized this by trying to make it bi-directional data-wise (your changes from Omea are reflected in Outlook and vice-versa). However this doesn't work for Newsgroups and RSS feeds for example.

Now the real questions is how should I use it? As a search tool? But then it's a bit heavy to be left sitting idle on my desktop. And it's too heavy to start it on demand (it currently takes 30 seconds to 1 minute to start - I'm sure it'll be improved in the future). As a productivity tool? Possibly, although it's missing some of the features I use in my specialized tools. For example:

• I use the "Reading pane - Right" view of Outlook which gives me 3 vertical panes next to each other. Once you get used to this, it's hard to go back. I'm told this will be in the next version of Omea Pro.
• I use NewzCrawler which let me see only feeds which have unread items in them, feed by feed (I don't read all blogs at the same speed).

Conclusion

I currently don't think I'll be able to keep Omea open all the time as it's too heavy to have both Outlook and Omea open at the same time. Also, I don't like the fact that I have to stop using my favorite Feed reader (NewzCrawler). I really do not want to manage 2 feed tools and tell each one which feeds I have already read. Of course, I could simply not use the feed feature of Omea. However I feel that using Omea just for searching makes it loose a lot of its attraction. If I just want a search tool, I can use Goodle Desktop or Lookout (even if they are less powerful they are probably good enough for my daily needs).

I think the real challenge for JetBrains is to make the tool good enough in each domain (mail handling, feed reading, etc) so that it can be used instead of the dedicated tool. That means that you would use Omea for day to day activities and the specialized tool from time to time only when you need one of the power feature. Of course, this is a huge challenge for JetBrains and honestly I am not sure if it is achievable.

Anyway, the tools is promising and intriguing enough so that I'll follow the different builds to see how it evolves.

If you're using it, please drop me a note on how you use it and how you handle it vs the specialized tools. Thanks

Update 04/11/2004: I've just tried build 358 and I am extremely pleased to report that the memory consumption has decreased a lot: whereas it wasa before 300MB for me, it's now 140MB. Good job!

I have to say that JIRA 3's extensibility is great! JIRA can almost be seen as a full fledge foundation for developing project tracking applications, in the same spirit as Eclipse is a full fledge foundation for developing java applications (RCP). Both Eclipse and JIRA come with a default application using this API to demonstrate their power (the IDE for Eclipse, the issue tracker for JIRA). Note that the new plugin system in JIRA has several similiarities with the Eclipse plugin architecture. Of course, I'm sure JIRA still has a lot of ground to cover to expose a plugin API covering all domains of issue tracking (i.e. allowing to replace all parts of the JIRA issue tracker) but it's going in the right direction.

Here is a short tutorial on how to develop a workflow function plugin. Note that you should also check the Atlassian tutorial on how to develop plugins.

The source code is available here and the plugin jar is available here.

Setting up the project

Here's the directory structure I have chosen for my plugin. Please also note that I have used Maven to perform the build (extremely easy to setup as Atlassian is also using Maven and they have all their jars in a Maven remote repository on http://repository.atlassian.com).

A plugin is composed of several files (it is packaged as a JAR at runtime):

• A plugin descriptor (atlassian-plugin.xml )
• Java source files
• Velocity templates for the plugin UI (the *.vm files)

The project.properties file simply adds the Atlassian Maven remote repo to the list of repos searched by Maven to download dependencies:

maven.repo.remote=http://repository.atlassian.com,http://www.ibiblio.org/maven

The project.xml contains the required JIRA dependencies and the definition of resources to include in the generated jar. Here's an extract:

Generating the plugin jar is as simple as typing maven jar .

The Worflow Function plugin extension point

Here's what the atlassian-plugin.xml plugin descriptor contains:

What you have to understand:

• A plugin is made of 2 java classes: a plugin factory class (SendMailFunctionPluginFactory ) which is in charge of setting up all that is necessary for the execution of the plugin feature, and the plugin execution class (SendMailFunction ).
• A workflow plugin is expected to bundle 2 velocity template files: one for asking the user to input some data required by the plugin execution (this is the input-parametes velocity template, and one for displaying what the function will do. The later is visible if you click on a workflow transition in JIRA and then on the post-functions tab.

The Java API

The Plugin Factory class

Without further ado, here's the skeleton for the SendMailFunctionPluginFactory class:

public class SendMailFunctionPluginFactory extends AbstractWorkflowPluginFactory
    implements WorkflowPluginFunctionFactory
{
     public SendMailFunctionPluginFactory(FieldManager fieldManager)
     {
      }
 
     protected void getVelocityParamsForInput(Map velocityParams)
     {
      }
 
     protected void getVelocityParamsForView(Map velocityParams, 
          AbstractDescriptor descriptor)
     {
      }
 
     public Map getDescriptorParams(Map conditionParams)
     {
      }
}

Those 4 methods are called by JIRA itself:

• The constructor is called when you click on the "add" button to add the function to your list of post-functions. The FieldManager instance can be used to get issue fields meta-data (it does not contain any issue data as there's no issue associated with the function yet - This will only happen when the function is triggered by an issue transition).
• The getVelocityParamsForInput() method can be used to store some properties in the velocityParams map. These properties will then be accessible from the "input-parameters" Velocity template.
• The getVelocityParamsForView() method can be used to store some properties in the velocityParams map. These properties will then be accessible from the "view" Velocity template. In addition the descriptor parameter provides access to the data entered by the user in the input phase (these data are stored in the workflow data structure itself).
• The getDescriptorParams() method is the bridge between the data contained in the Velocity context and the data in the Workflow context. More precisely you put in there the code to extract the data that have been entered by the user in the Velocity context and you put the data in the workflow descriptor context. This descriptor context is the second parameter that is available in your getVelocityParamsForView() method.

Here's a look at the input-parameters velocity template:

As you can see, the variables groupEmails and individualEmails will hold the data entered by the user.

The Plugin Function class

Here's the code that implements the plugin feature (in our case the sending of the notification email):

public class SendMailFunction implements FunctionProvider
{
     public void execute(Map transientVars, Map args, PropertySet ps)
     {
      }
}

The execute() method is called by JIRA when an issue transition happens.

The parameters have the following meanings:

• The transientVars parameter holds useful data such as the issue that was modified. You get a referernce to the issue by calling transientVars.get("issue"); . It contains also other piece of data such as the comment entered by the user, etc.
• The args parameter holds all the data stored in the workflow context (aka the workflow descriptor). This is the data you have stored yourself in the getDescriptorParams() method explained above.
• I'm not too sure what the ps parameter is used for. I think it holds data related to the workflow steps but this needs to be confirmed. Anyway, you shouldn't need it in most cases.

Deploying and executing the plugin

Deployment is a simple as dropping the plugin jar in [jira install dir]/atlassian-jira/WEB-INF/lib for example (it may be possible to drop it in other classloaders but I haven't tried).

In the following image we can see how JIRA has automatically discovered our plugin and extracted information from the plugin descriptor to make them available at the right extension point in JIRA:

Here is the page (using the input-parameters Velocity template) to let the user enter data for our function:

Here is how our function is displayed (using the view Velocity template):

I hope you got a good feel of what's possible to do with the JIRA API.

Note: For those wondering, I am not affiliated with Atlassian at all. I simply happen to like their tools (JIRA, Confluence) and I like the spirit of their team.

Note: There is also an article and a discussion thread on TheServerSide about this inititative

Here's a new idea for an open source project I'd like to start. It could be called CCI (Container Client Interface). It would aim at providing a simple Java API to start/stop/configure/deploy Java containers (In the first version the goal would be to support J2EE containers). This Java API could then be used by lots of other projects (Ant tasks, Maven plugins, IDE plugins, Cactus, etc).

The inspiration comes from the Cactus project, which already provides an extensive API to perform such tasks. The goal is to refactor the Cactus Ant API, remove anything related to Cactus and make it a standalone project called CCI

Here's an integraton test showing what the API could look like:

    public void testStartWithOneWarDeployed()
    {
         Container container = new Resin3xContainer();
         container.setHomeDir("/apps/resin-3.0.8");
         container.setPort(8080);
         container.setInstallDir(new File("target/resin3x"));
 
         WAR war = new WAR("src/testinput/simple.war");
         URL pingURL = new URL("http://localhost:" + PORT + "/simple/index.html"); 
         war.setPingURL(pingURL);
 
         container.addDeployable(war);
 
         ContainerRunner runner = new DefaultContainerRunner(container);
 
         runner.start();
         assertTrue("Container not started yet!", new HttpUtils().ping(pingURL));
         
         runner.stop();        
         assertFalse("Container not stopped yet!", new HttpUtils().ping(pingURL));
     }

As you can see there are several main Objects/Interfaces:

• Container: This the main object that provides the API to start/stop the container.
• WAR: A WAR archive to be deployed in the container. Inherits from Deployable. There will be other types later on (WAR, EAR, RAR)
• ContainerRunner: Athough a container can be start ed without a container runner, the container runner provides advanced feature like starting the container in a different thread, verifying if the container is already started, waiting till the container is started, waiting till the container is stopped, etc.

Note that the ping URLs are the URLs that will be pinged by the ContainerRunner object to ensure the container has finished starting. Thus, after container.start() has finished executing you can be sure the container is started and the archives have been deployed and are ready for servicing requests. A better solution in the future will be to use JMX to ensure the container has finished starting. However, all containers do not yet support this feature.

A shorthand code version to start Resin 3.0.8 could look like:

        Container container = new Resin3xContainer("/apps/resin-3.0.8");
        container.setInstallDir("target/resin3x");
        container.addDeployable(new WAR("src/testinput/simple.war"));
        container.start();

This is just a taste of what he CCI API could look like. Let me know what you think!

• Ability to start the Abbot Costello editor using either jars defined in the POM or the jars from an already installed WebStart application.
• Ability to execute Abbot XML scripts through the Ant <junit> task
• Ability to execute Abbot XML scripts on an already installed WebStart application (end-to-end functional testing)

You can download the Maven Abbot plugin here.

Please note that this plugin was written jointly with Christian Blavier whom I thank very much for his help!

• Cactus v2
• Create a container OSS project that offers a Java API to configure containers, start/stop them and deploy archives. The idea is to extract the existing API from the Cactus project (which already supports these tasks for several containers: Tomcat, Orion, Resin, JBoss, WebLogic, etc). Cactus would be refactored to use this new project. This API would be nice for any other project that requires a container.
• Implement a Maven plugin for Clirr. Note: I have almost finished a first version.
• Implement suppor for history reports in Maven. For example it would be nice to get Clover reports over time, Checkstyle reports over time, Dashboard reports over time, etc. I'm currently thinking to consider reports as project artifacts and store them in the Maven repository. A history plugin could provide support to save/load them. Another solution (probably better) involves using a lightweight database (e.g. Hypersonic SQL) and considering this database file as the artifact.
• Continue adding support for RSS feeds to the existing Maven plugins, wherever it makes sense. The final idea is to offer RSS feeds to all project information so that project members can have their own Personal Project Information Portal (PPIP). Among the tools I like/use, RSS feeds are already available for: JIRA, Confluence/TWiki, FishEye, Maven Checkstyle plugin, Maven Changes plugin. Next in my list are: Maven Clover report, Maven Dashboard plugin, Maven PMD plugin, Maven XDoc plugin (for downloads).

In addition to this, I plan to continue fixing Cactus v1 bugs (albeit slowly...) and continue fixing bugs and adding enhancements to the Maven plugins I have started (changes, clover, abbot, etc).

If you're working on the same topics or if you're interested in contributing, please let me know!

When you're working in a team one of the most important aspect of development is to know what others are working on, what they are modifying. Indeed, in development everything is tied one way or another:

• if someone changes the database schema, I need to know
• if someone updates a build dependency to a newer version of a framework I need to know
• if someone changes a public API, I need to know
• if someone starts working on the same set of source files as mine, I need to know
• if someone modifies a best practice document, I need to know so that I can have a look at what's new

Basically, I need to know whenever some source file on which my work depend on is modified.

Of course, I could rely on verbal communications: "Hey Joe, I'm gonna work on this files for the coming 4 hours". Sure, that'll work too. However it's not a scalable solution (Imagine a project divided in 5 teams of 10 persons each) and it works only if the participants are acutely aware of what each developer is depending upon which is near impossible. In addition, if the team is distributed it becomes virtually impossible to communicate such small events in any efficient manner. There has to be a better solution.

And there is! There are actually 2 solutions I know of, one better than the other.

Solution 1: Diff emails on SCM commit

This first solution involves setting up a server-side hook in your SCM. Most SCMs supports executing a script upon source check-in/commit. Just write a script that sends an email containing the diffs between what the user is committing and what already exists in the repository. The reason for the diff is that we're interested by what's different and not by the full content. This is what is done in most open source projects. Here's an example:

The problems with this solution are:

• It's not very easy to let the developer choose finely what files/directories they want to monitor
• The diff information is received in your email, among all you other emails. It's good to reserve emails for actions that you have to perform. Here we're not talking about actions but about information. I believe a better channel for information is for example a RSS feed. It still comes to you but it does not clutter your other emails.

Solution 2: RSS feeds

This solution involves setting up some kind of server that will generate RSS feeds for the developers. Implementing this could be quite involved. Fortunately there's a tool called FishEye from Cenqua that does exactly this. It's an improved viewcvs. Most importantly it allows developers to generate RSS feeds on any resource (any directories and any files). Here's what the HTML view looks like:

Then here's what you get in your favorite feed reader (I'm using Newz Crawler here):

Clicking on the diff link generates the following view:

Note: Please note that FishEye currently only supports CVS. Other SCMs like Subversion are planned in the future.

Conclusion

It is relatively simple and very effective to set up a way to notify developers about changes happening in the repository. I would even suggest that not doing so is a "communication anti-pattern". In order for it to be the most effective possible, developers should not be "spammed" by tons and tons of diff information. Thus to prevent this spamming you should consider 2 options: using RSS feeds instead of emails and allowing developers to choose their feeds. There are obviously some feeds that should be more "mandatory" than others. For example, the one notifiying of database schema change, the one notifying of build changes and the one notifying of public API changes. It's important, as always, that there is a champion in the team, explaining to others the benefit of these feeds and how they should be best used. Otherwise the power of them may exist but it won't be harnessed.

Here are the slides of my 2 presentations:

• Enterprise builds (description).
• Case study: Distributed Agile Development (description).

Update: All TSSS slides are available here (registered attendees only).

Pattern Testing is a concept I started researching in 2002 on a big project at work. At that time, I was already using Checkstyle and PMD to perform verification of rules. However, I was finding them limited in several ways:

• They were mostly focusing on syntactic rules, whereas I was mostly interested in semantic rule checking. Checkstyle has improved a lot in this area over the past 2 years. However, even though there are some semantic rules , they are usually simple and not very "business"-oriented. Note that this is not to diminish Checkstyle which I find great and use on all projects.
• They were focusing on applying rules on a single file (using AST trees), whereas I was interested in applying rules that span several files (this is necessary for architectural/design checks). As a consequence writing pattern tests using Checkstyle is difficult.
• They were limited to static checks. I wanted to be able to say "ensure that no method call is passed a null parameter", "ensure that there are no more than 10 calls to the database per user case", etc

The result was the Pattern Testing project on SourceForge. It's implemented using AspectJ. It contains some pre-made Pattern Tests but more importantly it lets you write your own. There is Maven plugin that makes it easy to run any Pattern Test on any mavenized project.

The possibilities of Pattern Testing are endless.

Here's an example of a rule that says that we do not want to instantiate business classes (i.e. a class that extends BasicBC). That's because they are instantiated by factories/service managers:

public aspect NoNewOnFrmwkClassesPatternTest
{
     declare error: 
       call(com.some.package.BasicBC+.new(..)) : 
         "Do not instantiate a business class directly";
}      

I used Pattern Tests for some time on that project at work and then I changed job. I was still interested in the concept but with my other open source involvements I didn't take the time to work on it as much as I wanted. Thus it became a bit abandoned. However, these past weeks, I have found renewed interest in the concept for several reasons:

• I was contacted by Matt Smith who was interested in taking the lead of the Pattern Testing project. I gladly handed this over to him and he's now the one driving the project
• I have started working on Cactus2 and as the Cactus 2 architecture is based on AOP, the concepts are very close.
• On the new project I am on, we're starting to want to automate our architectural/design checks and some teams have initiated using Pattern Tests.

Are any of you using Pattern Testing too?

It explains what are the day to day activities of an offshore coach (I prefer the word "coach" to "project manager") on a collaborative development project.

Here are some tasks that we (we are 2 from Pivolis working on this project) are performing on this project:

• Administrative tasks:
• Arrival/Departure administration (Secure id card, Star Team account, Test Director account, Wiki account, JIRA account, mailing lists accounts, etc)
• Organisation of visits (identification of requirement, dates, agendas, people) in both directions Paris -> India and India -> Paris
• Visa handling
• Infrastructure for offshore:
• Infrastructure coordination + improvements (monitor performances, tune tools) - with help from BNPP infrastructure team
• Organisational:
• Weekly technical and management conf calls with all the teams (dev team + integration team + tech team) + follow up from these calls
• Management meetings
• Technical meetings
• Weekly management level meetings
• Communication facilitators + related communication improvements
• Offshore team selection (senior members mostly)
• Offshore training programme improvements
• Crisis escalation investigation + handling
• Accompany Paris members during offshore visits, ensuring success of agenda/visit.
• Methodology/Expertise:
• Definition of Project Development Process with “offshore in mind” and continous improvements (short iterations introduction, usage of JIRA for detailed plannings, acceptance criteria for code delivery, wiki setup, etc).
• Work inside the teams to implement the collaborative development process.
• Quality suggestions + improvements (testing strategy / build improvements for quality + measurements)
• Expertise on build (Maven) + continuous build + testing + collaborative development processes
• Architecture + implementation of regression platform

• Enterprise builds: This session covers the different architectural concepts required for continuous builds of large projects. We show several strategies and explain how to implement these concepts with different continuous integration tools like Ant, Maven, CruiseControl, etc. The goal is to show what it takes to succeed in implementing an enterprise build. We cover concepts such as environment-dependent builds, binaries dependencies vs source dependencies, snapshot vs reference versions, continuous build strategies, where do tests fit in the build cycle, database strategies for continuous build, team organization related to build, etc.
• Distributed Agile Development (case study): This session will present Pivolis return of experience on a large scale J2EE banking project (approx. 100 persons in development team) developed collaboratively from 2 locations (France and India). We will discuss how we addressed the challenge of migrating from a collocated development setup to a collaborative distributed one. We will also detail how the project was made more agile in the process. We will cover the pitfalls that happened and see how they were resolved.

There's also a nice wiki available for the event.

Once the event is finished, I'll post my slides on this blog.

This will make it easier for them to improve it and keep it in sync with StatCvs-XML. This is possible thanks to the plugin:download feature of Maven 1.0 rc1/rc2 which allows users to easily install some external plugins. It is also possible to reference a plugin in your own project.xml as a dependency of type plugin .

The really nice part is their open architecture:

It's nice for the following reasons:

• There can be as many parsers as there are languages, script languages, etc. All data goes to the same database and this data is available to all analyzers. This means that it's possible to define analyzers that act at the level of the full application. For example, you can define some architecture pattern validation. This is the same idea I had some time ago with the PatternTesting project, using AOP to write these rules (I was relying on the Javac parser to perform the source code parsing for me).
• It allows to write browsing tools that show the full flow of the application, from configuration files to database tables

Compare this to existing open source tools (or not open source for some) like Checkstyle, Clirr, PMD, Findbugs, jcoverage, Simian, Clover, etc. They all use their own parsers and more important their own persistence format for storing the parsed data. Even though they almost all offer an XML report format, it's already processed data (as opposed to raw parsing data). Thus it is very hard to reuse the information from these tools to offer a higher level integrated view. It's also a lot of work for these tools to develop all the necessary parsers.

What would be really nice is the creation of an open source project that does the following:

• Define a common storage model (database schema, XML schema, other). It should allow to store historical data too.
• Define a common API to store parsing data
• Define a common API to query parsing data

If such open source project existed, we could have parser projects which could parse Java code, .Net code, Hibernate configuration + code, Struts configs + code, AspectJ Aspects, etc. These parsers would user the common API to store parsed data in the common storage format, thus making the data available to other projects, such as the one mentionned above (Checkstyle, PMD, etc). These projects woud be analyzers and would use the common API to query parsed data and generate reports for their domain of work.

With this, we could build nice analyzers that could completely check for full architecture best practices, across technologies and across frameworks.

Do you think such a project would work?

Note: Maybe CAST should open source this part (storage model) and keep as closed source their added value Parsers and Analyzers? :-)

They have accepted to run it on Cactus. It's very nice. The things I like particularly:

• RSS feeds on commits, at all directory levels. For example you can subscribe to the feed that monitors changes to the xdocs files if you're interested in documentation updates.
• Annotations (for e.g.: http://fisheye.thecortex.net/viewrep/jakarta-cactus/build.xml?r=1.69)

A year ago, I spent some time looking at Alice Bot. It's a bot that allows discussing with human beings. It has won several times the Loebner prize, which means it gets as close as possible to imitating a human person. The bot is controlled by XML files written in AIML.

The idea that I had at that time was to use it as a live FAQ answerer for the Cactus project. I never came to do it though... need more time...

I've been reminded of this idea by some posts on the Cactus mailing list. I'd venture that at least 50-70% of the the questions asked have already been answered in some form in the past, are available in the Cactus documentation or are easy to answer by some generic rule. I think it could be fun to train a bot to answer these questions. For those that the bot couldn't answer, the user would be redirected to the mailing list where Cactus experts would find the answer and train the bot so that it gets better next time.

It should be possible to have rules that would generate the following kind of dialogue:

• [user] I'm getting an error when running Cactus tests
• [bot] Is the error happening on the client side or the server side?
• [user] I don't know
• [bot] Could you turn logging on and see if the error appears in the client side log or the server side log?
• [user] how do I turn logging on?
• [bot] See the Cactus logging configuration page
• [user] It's happening on the client side
• [bot] Is the error hapenning during the HTTP connection?
• [user] yes
• etc...

It would be cool! My only worry is that it would take too long to create the AIML files. But maybe not. Any experience?

What would be even better would be that the bot would learn from the mailing list posts itself. However, that's probably science fiction at this point in time...

Note: The image was borrowed from Alicebot.org

Whenever a blog entry is updated by its author it appears in bold (indicating a change) and in italic (indicating that it's an update). However I have no clue what modifications were done to the entry and most of the time I don't have enough energy to read again the full entry trying to figure out by myself what has changed.

What would be really nice would be that the RSS reader shows a diff, highlighting the changes (as it's done for CVS commit diffs).

Features in this version includes:

• Ability to automatically generate the Jetty XML configuration file
• Ability to start Jetty
• Deployment of WAR defined as dependencies and tagged with the jetty.bundle property
• Deployment of the current project if it's a WAR project

Note that this plugin has been tested with Maven 1.0-rc2. Using it with another version of Maven is at your own risks! :-)

You can download the Jetty Maven plugin here.

• Reason 1: a change in the project itself is causing the failure
• Reason 2: a change in one of the project's dependencies is causing the failure

Solving case 1 is relatively easy. However, solving case 2 is usually quite difficult. How could we make it easier?

Here's a proposal for improving continuous build tools:

When a project goes from a "success" build state to a "failure" build state, perform a source diff (using CVS or whatever SCM used) between the 2 dates on both the project itself + all its dependencies. Then generate a report showing all the changes.

This should allow to get a clear view of all the things that changed and that led to the build failure. This strategy seems especially well suited to continuous build systems as these builds are executed often and thus the differences between 2 builds should be small enough to get a clear picture.

Has anyone done this already? Does it work?

Note: I have suggested this idea to the Gump project. I would be very eager to see how it works out.

Implementation details:

• If your SCM is too slow to perform large diffs over several revisions there is still hope! Save the last build checkout and perform a file diff against the current checkout and the last saved one. If you're using CVS or an SCM that supports putting metainformation in source file (like $Id:$) then you'll also get to know who made the change. Otherwise, for each file change you'll need to query your SCM to know who made the change.
• I think it might be interesting to target the build failure emails to the list of people who have made changes since the last build + at least one person from the failed project's team.

Limitations:

• Say we have project A which depends on project B which depends on project C. If project A fails because of a change in project C, then we won't know it unless we're doing diff on transitive dependencies. But that might be too long. I think a good strategy would be to only generate diff on the first level dependencies. Then possibly provide a feature to generate transitive dependencies on demand (but not at at every build failure - it'll be too costly I think).

In the meantime, please welcome Matt and let's wish him the best of success with the PatternTesting project!

I've just realized that collaborative offshore and EJB have a common point: they are both using a distributed model. By collaborative offshore, I mean teams developing on both sides (onsite and offhsore) and interacting continuously to build a system.

I've been working on 2 big offshore projects so far for the past 2.5 years (working with an Indian partner) and I've found that there is an organizational model that does not work well: onsite people directly managing developers (Figure 1). In the same way, calling directly Entity Beans from the client side is a bad practice because 1) it involves a lot of network round-trips and thus is inefficient and 2) it does not allow changing the implementation without affecting the client (Figure 2).

Figure 1: Onsite Project lead managing directly offshore developers

Figure 2: Client calling Entity beans directly

What is the solution we have used for EJBs? Answer: introduce a facade (Session Bean) which "manages" the underlying components (Figure 3). I've found that it is the same with collaborative offshore: there is a strong need to always introduce a local Project Lead (Figure 4).

Figure 3: EJB client calling a facade

Figure 4: Onsite Project Lead interacting with a local Project Lead

It probably sounds very obvious but very often there is an initial tendency by onsite managers new to offshore to directly manage offshore "resources" (in order to reduce support costs). So far, whenever they've tried, it has failed (although we told them it wasn't a good idea but it seems some people need to see it by themselves to believe... ;-)).

I wonder if there'll be a time in the future where our communications skills will be so great that it will allow direct managment across the wire. Probably... but this is still in the future...

Has this also been your experience?

• Do not mix unit tests independent of the environment (i.e. where interactions with the environments are stubbed/mocked) with integration unit tests (IUT). They have to be separated and put in different different in the SCM. The reason is that the 2 kinds of tests do not support the same execution workflow. More below.
• Have a database build project (in the sense of an Ant or Maven project) in your SCM. This is extremely important. The goal of this project is to provide the following build targets/goals:
  • create-schema: create the database schema from the ground up (in the database specified as properties)
  • load-static-data: loads static data (i.e. read only data)
  • load-minimal-data: loads a functionally minimal set of data. It should contain all data required functionally but only 1 or a few entries of each type. It's not supposed to reflect the state of the database when in production.
  • load-full-data: loads a full set of data as expected in production.
• Put the database data (minimal + full sets) in your SCM as flat files (by opposition as keeping the data live in the database). The reason for this is because:
  • you get automatic notification of data changes by using your SCM send-email-on-commit feature that all good SCM have
  • it is build-friendly and allows automated and controlled builds
  • it is controlled, i.e. you know what you're doing with your data, who is modifying them, you can revert if need be, etc
• Here's the workflow for executing IUT or functional tests. For each project and before the test suite runs:
  • execute database:create-schema
  • execute database:load-static-data
  • execute database:load-minimal-data

  Then, each test should also have the opportunity to load data in its setup (using DBUnit or similar). This is required for example to test special cases where the database is missing some required data and we wish to verify the exception handling part of the code. It is also required if the test requires more than the minimal data set (although that should be relatively infrequent).

  Note that the tests can also be ordered to save some database load time. Although not the best strategy I've found that this was sometimes required on some projects with complex database models.

On the other hand, here are strategies that have not been working so well for me in the past:

• Have a live database where developers can directly update data. The problems encountered were:
  • it is not controlled. You do no know who's putting data and what is being modified. You cannot easily revert a change
  • it's difficult with distributed teams as you need to set up a replication mechanism. The problem is that often developers update their local database and forget to update the master database which leads to lots of build failures. The solution exposed above does not suffer from this problem.
  • It's hard to sync everyone on the exact same set of data. Some minimal data + variations works best.
• Do not provide minimal data and let developers write from scratch the data they require for their tests and load these data before each test. This does work for small projects with simple database models but not for complex ones. There's really the need for a minimal data set.

Is that also your experience?

Here's what I feel is wrong with Starteam:

• No "clean checkout" option. That is, if a file is deleted from the StarTeam repository, even if you perform a checkout all, the deleted files will not be removed from your local working copy. Actually it is possible but only through the command line interface.
• No ability to send email diffs on commits (using tools like CVSSpam)
• No nice IDE integration such as the CVS integrations we can see in IntelliJ, Eclipse, NetBeans, JBuilder, etc. More specifically the ability to see exactly what files are not in sync with the repository.
• Limited integration with the majority of development tools: limited integration in Maven (BTW that's because it's been developed on this project itself by Emmanuel Venisse that there is limited integration in Maven!), no JIRA integration, etc
• No JIRA integration. It seems there's a nice CVS integration which allows to link source code to issues resolved by entering the issue number in CVS commits. As we're using JIRA we could use this!
• No possibility to run an Ant or Maven build every time there's a commit (a la Damage Control). That would allow our build to be in a better shape.
• Starteam is very slow on WAN links. It may be due to our project policy to use locking on files. In any case if we had CVS we wouldn't have used this locking which is hampering productivity.
• No windows explorer integration. To use Starteam you need to open yet another GUI application and perform operations from there (unless you use the command line but nobody is using it here). CVS has a nice TortoiseCVS client.
• With Starteam a major problem for our build is that people forget to check in directories. No wonder as the Starteam GUI client does not show at all new directories!
• Starteam is quite expensive and as a result we have only a limited number of fixed licenses. Anyone using floating licenses gets disconnected every few minutes. Very annoying.
• Starteam admin seems more complex than CVS's. We have often had problems of database in the past year.

Any more? Any inaccuracy in there (I'm sure there are as I am biased!)?

Some of my Octo workmates are developing a java framework for generating captchas called JCaptcha. What I find interesting is one possible use of captchas: preventing spam. More specifically the idea would be to use captchas to prevent blog spam. It means that people who enter blog comments would need to be humans. What I don't know is whether blog spam is being done manually by individuals or if it's automated. In any case this solution will prevent automated blog spam which is a good first step!

The JCaptcha project has just released a beta version. I guess one next step could be the creation of a MoveableType plugin. Then I would hope to convince Bob to let us try it on Codehaus blogs :-)

Update 18/04/04: It seems that James Seng beat me with this captcha idea. Not only the idea but also an MT implementation. In addition, he's created yet another MT plugin for preventing comment spam by implemeting an MT Bayesian filter.

However, even today in 2003, I still think it had some nice ideas that I would like to see in existing and future IDEs. Here are some ideas about this Jini IDE:

• Each module would be a Jini service. Examples of modules are: javac compiler module, RMI compiler module, java editor module, source repository module, java execution module, junit execution module, etc.
• It would be lightweight. It would be able to bootstrap with a minimal jar containing only the "microkernel" (+ possibly a module cache manager). Thus you could move from one machine to another easily. To install it, simply click on a browser link and download this minimal core. The rest will be downloaded as need be when you need the modules.
• As each module is a Jini module, there would be 3 possibilities to implement a module:
  • The Proxy contains only local methods and there are no communication between the proxy and the back-end service,
  • The Proxy is a “smart proxy”, i.e. there are both local and remote methods which communicate with the back-end service,
  • The Proxy is only a client stub, all the methods are remote
• It would be completely distributed. For example, the compile menu of the IDE would list all compiler module implementations the IDE has been able to discover when contacting the different Jini lookup services.
• It would be self-healing: if a module is no longer available on a given server, another replacement will be automatically discovered (by the magic of Jini leases).
• By using Jini leases, the IDE would support hot-patching/uninterrupted services
• It would be secure using Jini security for modules. This will allow to support both open source modules and commercial modules.
• What would be nice would be to have a repository service which would automatically save edited code on the server side (in a user-private zone). This would enable remote building. Developers on the move would be able to get their environment set up rapidly on any machine. Same if you wish to share environment with someone else, etc.
• It would be completely modular with caching done on the client side to improve performances. The modularity should allow the creation of module repositories on the web. It would also allow creating IDE "a la carte".

Of course, this is a bit utopic as we would need to overcome several difficulties:

• definition of standard module interfaces. However lots of work has been done there already by Netbeans and Eclipse and that could be reused.
• there would need to be some module certification tests to ensure a module is properly coded, does not hog the IDE, plays well with others, etc

Of course, this IDE of the future could be implemented in a technology other than Jini (P2P, Web services, etc). I still believe Jini is way ahead of web services but they are catching up slowly on security, transactions. To my knowledge there's still no notion of "leases" in web services, nor of code that moves around the network and can execute locally.

What do you think? Is that Jini-like IDE something you would also like to see in the future?

Update: This blog entry has been reposted on TSS (several interesting comments).

Anyway, I've found that there are different kinds of projects. Some where the build is taken very seriously and a build-aware mentality quickly spreads and others where people do see the value of the build but have more difficulties taking it very seriously (leading to lots of build failures).

One idea that I've had is to use a physical artifact to represent a build success or a build failure. People like to see and touch things. Doing some research I've found the Ambient Orb which seems close enough to what I have in mind:

(Image stolen from ThinkGeek)

The idea is that the orb will turn more and more red depending on the number of projects that failed to build during the past cycle.

I have yet to buy one and try it but I like the idea. Has anyone done this already? Can it be done easily with the Ambient Orb? Are there devices other than the Ambient Orb on the market (for example, I don't need the wireless radio network connection at all especially as I live in France)?

Update: This blog entry has been reposted on TSS.

• Rationale
• Architecture choices
• High level architecture
• Cactus test case example
• Detailed design
• Challenges

Rationale

Why a new architecture? Several reasons:

• The existing architecture is restricted to testing Servlet components (and its variations: Taglibs, Filters, JSP). We've tried to create an SPI so that implementations for other containers can be written but it is not possible with the current architecture.
• We'd like to make Cactus the de facto tool for performing integration unit testing (aka in-container testing) for any type of component in any type of container. See figure 1.
• We'd like to make it easy for others to create Cactus extensions. This is not currently possible with the existing architecture.
• We'd like to maximize the reusability of other testing tools. For example, instead of implementing in Cactus the HTTP layer that calls the server side, we'd like the user to use his favorite tool (e.g. HttpUnit). For unit testing Message-Driven Beans, the user will be able to use his favorite JMS injector (e.g. Commons Messenger), etc. This will allow leveraging all the features in those tools (for example, support for HTTPS in HttpUnit, support for Cookie handling, etc).
• We'd like to standardize on a server side interception mechanism instead of inventing our own.

Figure 1: Scope of Cactus v2

Architecture choices

These are the high-level architecture choices on which Cactus v2 will be built:

• An AOP framework for server-side interception. There are 2 possible contenders: AspectJ or AspectWerkz. We are currently favorizing AspectWerkz because it allows to write test cases in Java. AspectJ extends the Java language and requires strong tool support. It will change with the advent of JDK 1.5 as the JDK will become meta-data compatible and Aspect will probably take advantage of this. However, it will take several years before everyone is on JDK 1.5 and we need a solution before this.
• Cactus v2 will continue to be a JUnit extension. A Cactus v2 test case will be a JUnit test case and thus any JUnit test runner will work (provided the server has been started and the components and tests deployed).

High level architecture

The Cactus system is composed of 3 parts (see figure 2):

• A Cactus test case, which is a combination of a JUnit test case (with testXXX() methods executed on the client side) and aspects used on the server side to perform interception and/or validation on the server side (see figure 3). More specifically, 3 typical uses cases for these aspects are:
  • Intercept the call to the component under test and redirect the flow of execution to a specific method to unit test it,
  • Prevent the flow of execution to call some subsystem. For example, stop the flow of execution before it goes to the database and instead return canned values.
  • Perform asserts to verify server-side expectations. For example, verify that the Servlet HTTP Session contains such and such values after executing such method, verify that the Database connection is closed as many times as it is open for such and such use case, verify that the number of SQL queries is below such number (e.g. less than 10 SQL queries per use case), etc. These are server-side expectations.
• A Cactus runner to execute Cactus tests automatically. This involves starting the container, deploying the application and tests in the container, starting the tests and stopping the container.
• A Cactus framework to support starting a test case on the client side and continuing it on the server side, and also to support transferring test results from the server side to the client side so that results can be displayed in the executing JUnit test runner. This framework also contains helper aspects and classes to help write test cases.

Figure 2: High-level architecture

Cactus test case example

Here is an example of a typical Cactus test case using AspectWerkz 0.9. Please note that this example is a work in progress and is non-functional at this stage. We're also working towards simplifying the syntax for test case writers:

package org.apache.cactus.sample.servlet;

import java.util.Hashtable;

import javax.servlet.http.HttpServletRequest;

import org.codehaus.aspectwerkz.attribdef.Pointcut;
import org.codehaus.aspectwerkz.attribdef.aspect.Aspect;
import org.codehaus.aspectwerkz.joinpoint.JoinPoint;
import org.codehaus.aspectwerkz.joinpoint.MethodJoinPoint;

import com.meterware.httpunit.GetMethodWebRequest;
import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebRequest;
import com.meterware.httpunit.WebResponse;

import junit.framework.TestCase;

public class TestSampleServletAspectWerkz extends TestCase
{
     /**
      * Intercepts Servlet's doXXX calls and instead redirect the flow of
      * execution to the {@link SampleServlet#getRequestParameters} method to
      * unit test.
      * 
      * @Aspect
      */
     public static class GetRequestParametersTestAdvice extends Aspect
     {
          /**
           * @Execution * *..SampleServlet.do*(..)
           */
          Pointcut interceptServlet;
          
          /**
           * @Around interceptServlet
           */
          public Object catchGetRequestParameters(JoinPoint joinPoint) 
              throws Throwable
          {
               MethodJoinPoint jp = (MethodJoinPoint) joinPoint;
               SampleServlet servlet = (SampleServlet) jp.getTargetInstance();
               Hashtable params = servlet.getRequestParameters(
                   (HttpServletRequest) jp.getParameters()[0]);
               assertNotNull(params.get("param1"));
               assertNotNull(params.get("param2"));
               assertEquals("value1", params.get("param1"));
               assertEquals("value2", params.get("param2"));
               return null;
           }
      }
 
     /**
      * Test {@link SampleServlet#getRequestParameters} by calling the server 
      * side using HttpUnit. On the server side, our aspect will kick in and
      * the {@link GetRequestParametersTestAdvice#catchGetRequestParameters} 
      * test method will be called to unit test our method.    
      */
     public void testGetRequestParameters() throws Exception
     {
          WebConversation conversation = new WebConversation();
          WebRequest request = new GetMethodWebRequest(
              "http://localhost:8080/test/SampleServlet?param1=value1&param2=value2");
          WebResponse response = conversation.getResponse(request);
      }    
}

We would like to be able to write the following (not yet supported by AspectWerkz but we've had commitment from the AW team that they will make modifications to support it! :-)). The difference with the previous sample is the removal of the inner aspect class + the typed poincut interception.

package org.apache.cactus.sample.servlet;

import java.util.Hashtable;

import javax.servlet.http.HttpServletRequest;

import org.codehaus.aspectwerkz.attribdef.Pointcut;
import org.codehaus.aspectwerkz.joinpoint.JoinPoint;
import org.codehaus.aspectwerkz.joinpoint.MethodJoinPoint;

import com.meterware.httpunit.GetMethodWebRequest;
import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebRequest;
import com.meterware.httpunit.WebResponse;

import junit.framework.TestCase;

public class TestSampleServletAspectWerkz extends TestCase
{
     /**
      * @Execution * *..SampleServlet.do*(..)
      * @And @Target(SampleServlet)
      * @And @Args(HttpServletRequest)
      */
     Pointcut interceptServlet;
         
     /**
      * @Around interceptServlet
      */
     public void catchGetRequestParameters(SampleServlet servlet,
         HttpServletRequest request) throws Throwable
     {
          Hashtable params = servlet.getRequestParameters(request);
          assertNotNull(params.get("param1"));
          assertNotNull(params.get("param2"));
          assertEquals("value1", params.get("param1"));
          assertEquals("value2", params.get("param2"));
      }
 
     /**
      * Test {@link SampleServlet#getRequestParameters} by calling the server 
      * side using HttpUnit. On the server side, our aspect will kick in and
      * the {@link #catchGetRequestParameters} test method will be called 
      * to unit test our method.    
      */
     public void testGetRequestParameters() throws Exception
     {
          WebConversation conversation = new WebConversation();
          WebRequest request = new GetMethodWebRequest(
              "http://localhost:8080/test/SampleServlet?param1=value1&param2=value2");
          WebResponse response = conversation.getResponse(request);
      }    
}

Detailed design

The detailed design of Cactus v2 is shown in figure 5 below.

Figure 5: Detailed design

It works as follows:

1. The Cactus tests are started by a JUnit Test Runner (any JUnit Test Runner).
2. The Cactus framework intercepts the JUnit call to the test case runBare() method. It checks if a listener socket has been set up. If not it sets up one. It passes the test name to it (so that the server side can later on find out what test is currently being executed),
3. It calls the test case testXXX() method. In this method the test case writer has written the logic to call the server side (using any existing framework; for example HttpUnit for calling an HTTP service),
4. The flow of execution reaches the application to test on the server side,
5. Somewhere during the execution of the application, the test aspect defined by the test case writer kicks in. Before that aspect is executed, the Cactus framework intercepts it,
6. The Cactus server side interceptor then calls the listener socket set up in step 2 to get the name of the test being executed (the test that was started on the client side). It checks if the aspect matches the current test,
7. If the aspect matches, its advice is executed, performing whatever logic the test case writer has put in it,
8. Before the call returns to the client side, the Cactus server side interceptor calls the socket listener to pass to it the server side test result (it passes to it any exception raised on the server side; for example AssertionFailedError exceptions),
9. After the testXXX() method finishes its execution and before the test result is communicated to the JUnit Test Runner, the Cactus client side interceptor verifies if any error has been reported by the server side execution. If so, it rethrows the server side exception to the JUnit Test Runner. Otherwise it lets the result of testXXX() bubble up to the JUnit Test Runner.

Some additional comments/ideas:

• If one of the catchXXX() methods is not called, it should result in an AssertionFailedError being raised. This is to prevent not executing server-side test code without knowing it. As we are using interception, I guess it's easy to make a mistake when defining the join point and thus we need this safeguard.

Challenges

The following challenges await us:

• Being able to make the Cactus test case easy to write for test case writers,
• Make the execution of Cactus test case easily executable. Runtime code weaving would be nice but is not supported by old JVMs. We will probably have a mixed model as is being supported by AspectWerkz.
• Find out if integration with Chad's VirtualMock is possible/desirable.

Please challenge us to improve our design! :-)

Disclaimer: Please also note that, at this point in time, this architecture and ideas are only mine and do not represent (yet!) the official view of the Cactus project. I am proposing it to the Cactus project members.

If you're interested to discuss a bit about unit testing, JUnit, TDD, Cactus, Maven, Ant, open source, Apache/Jakarta, etc come and join me!

For those in other countries, sorry! I don't think Manning will pay for the trip. That's unless JUnit in Action becomes a bestseller of course! You know what you have to do! :-)

I'd like to expand its goal and make it a framework for building in-container testing solutions. Cactus would still offer an implementation for J2EE component testing but it will also open up an API for plugging other implementations. Some ideas are shown on the diagram below.

For this to happen, the core helper classes will have to be separated from the HTTP protocol implementation and the existing Cactus TestCases. 2 SPIs will appear:

• one for plugging in different protocol implementations (RMI, JMS, etc). Currently Cactus provides the HTTP implementation.
• one for plugging in custom test case implementations (still looking for a good name for these). Currently Cactus provides the ServletTestCase, FilterTestCase and JspTestCase.

Moreover, the Cactus integration modules (aka front-ends) will also need to provide clearly-defined extension points to help automate the whole process of starting the container, deploying components, running the tests and shutting down the container.

I'm currently working on the Cactus code to make the 2 SPIs surface. The first test drive of these new SPIs will be to implement support for EJB TestCases.

Here are some resources about JIA:

• JIA web site.
• Table of contents.
• Sample chapters.
• Source code. JIA source code has been donated to open source under an Apache license.

Reviews:

To use it:

• put the extension jar in front of the easymock jar in your classpath (it works with Easymock 1.0 only, not with the new 1.0.1b which has modifications for extending Easymock). It overrides the MockControl class by a new one accepting a class as parameter as shown below.
• add cglib to your classpath (I've tried it with version rc2-1.0).
• add bcel to your classpath (I've tried it with version 5.1).

Here's an example. First let's start by the class we wish to mock:

package test;

public class Calculator
{
     private int amount;
     
     public Calculator(Integer amount)
     {
          this.amount = amount.intValue();
      }
     
     public int compute()
     {
          return this.amount;
      }
}

Here's now the class we wish to test (it uses the Calculator class):

package test;

public class Account
{
     public int computeBalance(Calculator calculator)
     {
          return calculator.compute();
      }
}

Now, here's our test of computeBalance . Notice that we are mocking the Caculator class:

package test;

import org.easymock.MockControl;

import junit.framework.TestCase;

public class AccountTest extends TestCase
{
     public void testComputeBalance()
     {
          Account test = new Account();
          
          MockControl control = MockControl.createControl(Calculator.class,
              new Class[]{Integer.class}, new Object[]{new Integer(5)});
          Calculator mock = (Calculator) control.getMock();
          
          mock.compute();
          control.setDefaultReturnValue(10);
          control.replay();
          
          int result = test.computeBalance(mock);
          assertEquals(10, result);
      }
}

Note: I don't know how to instantiate a constructor taking primitive types. Don't know if it is supported.

That's nice. However, there are a few drawbacks I can see:

• It doesn't encourage refactoring and creating interfaces (which is the right way to go in most cases). However, it can still be useful in some cases like when mocking third party classes which do not have interfaces (and there are quite a lot of them, especially in the JDK...)
• It doesn't work with final classes. Thus not possible to mock the JDK's URL class for example. It also won't work with private constructors of course
• It forces to know the constructor values when creating the mock
• It still doesn't help in cases where it is no easy to introduce the mock in the class under test. In that regards, it's less powerful than interception (a la AOP).

So it's not a silver bullet but it will certainly help in cases where you have no control over the sources (like when using third-party libraries.

The e-book edition will be available sometime this week.

22/10/2003 update: the ebook is now available.]]> Personal events vmassol 2003-10-19T20:27:33+00:00 http://blogs.codehaus.org/people/vmassol/archives/000170_using_rollbacks_for_unit_testing_database_code.html Let's imagine you wish to perform unit test for code that calls the database. Let's also imagine that you want to test in integration, i.e. verify that the SQL query does actually goes to the database and returns database data.

The traditional approach is to:

1. Preload static data before the test suite runs
2. Load specific test related data in the database before each test. A variation is to restore the modified data at the end of the test

There are several disadvantages to point 2:

• Finding out the specific data that needs to be set up for the test is time-consuming and you often need to know the full database schema (and not only the domain you're working on). This is especially true for big projects
• There are referential integrity concerns that forces you to set up again data for a lot of other tables that the ones you are concerned with as they are linked through keys

One solution that our project team discussed yesterday is about using transaction rollbacks. It would work as follows:

• Before the test suite starts, load the database with full data (this is done only once and usually takes a good 5 minutes for complex projets)
• Before the test starts, start a transaction (in JUnit's setUp() method for example)
• run the test
• rollback the transaction to restore the database data in a pristine state

I was suprised to see no mention of this on the articles mentioning database unit testing (http://www.dallaway.com/acad/dbunit.html, http://www.dbunit.org/bestpractices.html). The reason may be that there are some glitches with this technique that make it impossible to use in practice... I'd like to know what you think? Have you done this before?

Notes:

• The code under tests should not create a new transaction as nested transactions are usually not supported

The Maven StatCvs plugin is a plugin for StatCvs-XML that generates CVS statistic reports.

Changes in this version:

• Migrated the plugin to use the new statcvs-xml jar (which generates XML xdoc output instead of HTML).

You can download the StatCvs Maven 2.0 plugin here.

One solution is to use AUT: AOP Unit Testing! (hehe... yet another acronym :-)).

Let's try that by writing a unit test using AspectJ for an EJB. Let's imagine we want to unit test the following createOrder() method:

package junitbook.ejb.service;

import java.rmi.RemoteException;
import java.util.Date;

import javax.ejb.EJBException;
import javax.ejb.SessionBean;
import javax.ejb.SessionContext;

import junitbook.ejb.domain.OrderLocal;
import junitbook.ejb.domain.OrderUtil;
import junitbook.ejb.util.JMSUtil;
import junitbook.ejb.util.JNDINames;

public abstract class PetstoreEJB implements SessionBean
{
     public int createOrder(Date orderDate, String orderItem)
     {
          OrderLocal order = OrderUtil.createOrder(orderDate, 
              orderItem);
  
          try
          {
               JMSUtil.sendToJMSQueue(JNDINames.QUEUE_ORDER, 
                   order.getOrderId(), false);
           }
          catch (Exception e)
          {
               throw new EJBException(e);
           }
          return order.getOrderId().intValue();
      }
 
     public void setSessionContext(SessionContext sessionContext) 
         throws EJBException, RemoteException {}
     public void ejbRemove() 
         throws EJBException, RemoteException {}
     public void ejbActivate() 
         throws EJBException, RemoteException {}
     public void ejbPassivate() 
         throws EJBException, RemoteException {}
}

Note that the nasty calls to OrderUtil.createOrder() and JMSUtil.sendToJMSQueue are static!

Our challenge here is to unit test this method in isolation from the rest. Here are the corresponding unit tests: one to verify it works when there is no exception (testCreateOrderOk ) and one to verify it also works when there is a JMS exception raised, for example (testCreateOrderWhenJMSException ):

package junitbook.ejb.service;

import java.util.Date;

import javax.ejb.EJBException;
import javax.jms.JMSException;

import com.mockobjects.dynamic.Mock;

import junit.framework.TestCase;
import junitbook.ejb.domain.OrderLocal;
import junitbook.ejb.domain.OrderUtil;
import junitbook.ejb.util.JMSUtil;

public class TestPetstoreEJB extends TestCase
{
     private PetstoreEJB ejb;
     
     protected void setUp()
     {
          this.ejb = new PetstoreEJB() {};
      } 
 
     public void testCreateOrderOk()
     {
          int result = this.ejb.createOrder(new Date(), "1234");
          assertEquals(1234, result);
      }
 
     public void testCreateOrderWhenJMSException()
     {
          try
          {
               this.ejb.createOrder(new Date(), "1234");
               fail("Should have thrown an EJBException");
           }
          catch (EJBException expected)
          {
               assertEquals("some jms error", 
                   expected.getCausedByException().getMessage());
           }
      }
}    

aspect TestPetstoreEJBAspect
{
     OrderLocal around():
         call(* OrderUtil.createOrder(..)) &&
         cflow(execution(* testCreateOrder*()))
     {
          Mock mockOrderLocal = new Mock(OrderLocal.class);
          OrderLocal orderLocal = (OrderLocal) mockOrderLocal.proxy();
          mockOrderLocal.matchAndReturn("getOrderId", new Integer(1234));
          return orderLocal;
      }
 
     void around():
         call(* JMSUtil.sendToJMSQueue(..)) &&
         cflow(execution(* testCreateOrderOk()))
     {
          return;
      }
 
     void around() throws JMSException:
         call(* JMSUtil.sendToJMSQueue(..)) &&
         cflow(execution(* testCreateOrderWhenJMSException()))
     {
          throw new JMSException("some jms error");
      }
}

Note: I highly recommend using Eclipse 3.0M3 and the latest AJDT plugin if you wish to run this code. The AJDT works great for writing AspectJ projects.

That's quite nice and it works nicely but as you have noticed the test code is neither very easy to write nor to read. What I would love is a dynamic language geared towards writing these kinds of unit tests! Could Groovy provide that? That would be soooooo nice!

I think that's an excellent idea! On most projects I've been on, writing good unit tests often comes late in the development. Usually people write functional or integration tests but rarely unit tests which test code in isolation from the rest. And once the design best practices have been decided and lots of code has been already written, writing unit tests is hard because it requires severe refactoring and design changes. The hard part is the introduce mock objects in the code under test. Some code uses statics, other instanciate domain objects instead of them being passed to it, etc.

Of course, extreme TDD practictioners would say that it is much better to refactor the code under test so that it become more flexible and better able to support change. That's true but that's difficult to practice for the majority of projects and difficult to apply if the project has not been started the TDD way.

What would be nice is to have a dynamic language geared towards introducing easily changes to the code so that it can support the mock objects approach. There is a unit testing framework out there that does almost this; it's called AgileTest from Polygenix.

Maybe Groovy could extend the ideas from AgileTest and become a generic dynamic language to easily insert mock objects (among other things)? Sprinkle this with an AspectJ-like syntax and you've got something quite nice!

I know I'm probably changing the original idea but what I'd love to have is a language purely geared towards unit testing

What do you think?

Note that more thorough examples are available from the NASA web site.

I would like to revive this idea and start drawing some concept maps about the JUnit ecosystem. I would like to have some of these maps in my JUnit in Action book but I would also like to set up a collaborative web site gathering all knowledge around JUnit using these concept maps.

This is all nice but the software I used then (IHMC Concept Map Software) is now very old (2001), the GUI is flaky and the software does not lend too well to distributed updating of the maps on the web (it uses a custom server which opens a specific TCP/IP port and thus does not easily allow updates behind a firewall, etc).

I need your help! My questions to you are:

1. Do you know how this concept has evolved? Is it still a hot topics nowadays?
2. Do you know of any software (preferable free) that would allow me to easily draw maps such as the one above and allows collaborative editing over the web?

Thanks!

There is a good article called Evolving Java-based APIs by Jim des Rivieres.

It's a complex subject that not a lot of persons are aware of and which is extremely important if you're a framework writer. Let me give one example.

Imagine you have a ServletTestCase public API class (i.e. users can use it) and it inherits from an AbstractTestCase class. This latter class is not supposed to be used by end users. However, it has some public API methods accessed through the ServletTestCase by the users.

Now, if you think you can safely refactor the AbstractTestCase class you're wrong! For example, if you wish to split the AbstractTestCase class in 2: AbstractClientTestCase and AbstractWebTestCase , the second one inheriting from the first, you're in for trouble...

Indeed, code that has been compiled with the first version of the framework will have a reference to AbstractTestCase in its .class files (provided it was using methods inherited from AbstractTestCase ). Thus when you bring in the new framework and put it in the runtime classpath of your other code, it will fail with a NoClassDefFoundError !

What it means is that preserving binary compatibility is difficult and is something that strongly impacts how you design your APIs. Using delegation has to be preffered over class inheritance as it will allow you to change the implementation without breaking binary compatibility.

That's what was new to me. So far I had been designing simply using the OO concepts and choosing the pattern the most adapted. Now, I've discovered that I also need to introduce the binary compatibility aspect in the picture and that it drives my design choices!

Evolving Java-based APIs is really difficult...

I was able to meet all my "heroes of the web": Mike Cannon-Brookes, Jason Carreira, Cameron Purdy, Cedric Beust, Gavin King, Howard Lewis Ship, Erik Hatcher and I also met some others great guys I had not met online before: Rod Johnson, Bruce Tate, Christophe Ney, Scott Ambler and a few others (sorry if I forgot you).

I also met Marjan Bace from Manning, who is editing my JUnit in Action book.

To top it all, I won a Tivo (not sure it works in France but we shall see).

I had the chance to speak in front of this great audience and it was cool. The talks I presented were:

• Unit testing J2EE applications. Here are the slides. All source code for the examples in the slides are available on the JUnit in Action code website on SourceForge (all source code was donated to open source under an Apache Public License).
• Building J2EE applications with Maven. Here are the slides. The source code for the Maven Everest project described in the slides is available here.

I was also invited to sit on the Open Source panel to answer the public's questions (along with Bill Burke, Christophe Ney, Gavin King, Mike Canon-Brookes and Erik Hatcher - Some pictures are available here) and finally I recorded a TSS Tech Talk on "Agile Offshore Software Development" (this is what we are doing at Pivolis).

I will not go in the detail of the sessions as it has already been well covered by Cameron and a few others (see Cameron's links).

I'm looking forward to the next TSSS. Would be cool if it were in Europe. Maybe we could also do it on a cruise boat next time!

PS: Mike, you were not the last one to blog about TSSS... :-)

• Unit testing J2EE applications: I will try to cover the full set of J2EE components using different techniques: MockObjects and Cactus. The hard part is understanding what each brings you and when to use them. I will also cover automating tests and continuous integration using Ant and Maven. I wanted to cover unit testing with an AOP framework but I'm not sure I'll have the time to research that enough... Most of the examples will be taken from my JUnit in Action book.
• Building a J2EE application with Maven: This talk will explain Maven and I will then do a step by step build of a J2EE application, dividing the full projet into Maven projects and proposing a strategy for organizing a J2EE project. I will also cover continuous integration with Maven.

If you're going there and want to meet up, leave me a message!

Note: I'm also looking forward to open a U.S. bank account there but I have no clue what paper I need to bring and how easy (or hard) it is. Any clue is most welcome :-)

• http://www.simongbrown.com/blog/index.jsp
• http://www.oreillynet.com/pub/au/113
• http://www.eugenebelyaev.com/begblog/general/000014.html
• http://blog.fuzzymagic.com/blog/rss/sean
• http://blogs.application-servers.com/blogs/page/doj&mag/ [in French]
• http://www2.theserverside.com/rss/theserverside-0.9.rdf

If you know others blogging from JavaOne, let me know and I'll add them.

Daniel Bradby has informed me he has just released an Eclipse plugin for Subversion.

I haven't tried it yet... If you have I'll be happy to know what you think.

Update 22/07/2004: There is now a TortoiseSVN wrapper plugin for Eclipse which seems quite nice.

The new plugin is currently located in Maven CVS and you need to build Maven from its sources to use it. In other words, it has not been released yet.

Note this new checkstyle plugin for Maven has only been tested with Maven 1.0 beta 10 and is not expected to work with other versions (but it may...)

Here's the book description:

A guide to unit testing Java applications (including J2EE applications) using the JUnit framework and its extensions, this book provides techniques for solving real-world problems such as unit testing legacy applications, writing real tests for real objects, automating tests, testing in isolation, and unit testing J2EE and database applications. Using a sample-driven approach, various unit testing strategies are covered, such as how to unit test EJBs, database applications, and how to unit test JSPs, and Taglibs. Also addressed are testing strategies using freely available open source frameworks and tools, and how to unit test in isolation with Mock Objects. Testing J2EE applications by running tests from inside the container for performing integration unit tests is discussed, as is how to automate unit testing in automated builds (such as Ant and Maven) for performing continuous integration.

The book will soon be part of the Manning Early-Access Program, which will allow subscribers to get access to all the book chapters as they are ready without having to wait for the book final publication.

The new version of the Cactus plugin now relies on the newest Cactus/Ant integration which has completely changed since version 2.0 of the Maven plugin. Thanks to the new Cactus/Ant, some features have also been added. The plugin now uses the new and Ant tasks (see the Cactus/Ant Integration page for more details).

Note that version 2.1-SNAPSHOT will never get released as 1/ a bug had been introduced and 2/ the direction is to use the Cactus/Ant integration which has been changed in Cactus CVS.

Please give it a try and report any issue on the Maven mailing lists/JIRA. You can find a read-made test application in here if you wish.

That said, lots of persons do find that writing xdocs is a pain. Moreover wouldn't it be nice if end users could easily contribute to the documentation?

Here's a solution:

• Use a Wiki for the project's web site. But not any wiki. Use a wiki that stores it's web pages in XML format, such as MoinMoin (Here's an example of generated XML. Use View Source for IE users).
• Then write a hook (script) in that wiki so that the modifications get saved to your project's CVS (or even better use CVS as the underlying storage for the Wiki). Question: is that directly supported by MoinMoin?
• (optional). Use a CVS syncmail script to send CVS commit diffs to the project's development mailing list. This allows everyone subscribed to see the changes to the documentation (and to the sources of course).
• (optional). Now that we have our XML doc sources saved in the project's CVS we can (if we wish) generate the docs in an HTML format different than the wiki we are using, for packaging them in the project's distirbution for example.

Nice, no?

On a related note, I've released version 0.09 of the MockObjects.com framework yesterday too. This release contains several improvements in provided Mock Objects but little modifications to the core, apart from the introduction of a new library: the Dynamic Mock API (or DynaMock for short). It is based on dynamic proxies (same as EasyMock but I think with a more powerful and cleaner syntax), which means the mocks are created at runtime.

Warning: This first cut of the DynaMock API is not stable at all. Actually, a big refactoring has happened in the MockObjects CVS and a better API has been developed (very similar but which fixes the quircks from the first cut). Thus, if you use this DynaMok API, be prepared to refactor your applications when version 0.10 is out!

Here is a short example using DynaMock 0.09:

[...]
import com.mockobjects.constraint.Constraint;
import com.mockobjects.dynamic.C;
import com.mockobjects.dynamic.Mock;

public class TestAdminServlet extends TestCase
{
    private Mock mockRequest;
    private Mock mockResponse;
    private HttpServletRequest request;
    private HttpServletResponse response;
    private AdminServlet servlet;
    
    public void setUp()
    {
        servlet = new AdminServlet();

        mockRequest = new Mock(HttpServletRequest.class);
        request = (HttpServletRequest) mockRequest.proxy();

        mockResponse = new Mock(HttpServletResponse.class);
        response = (HttpServletResponse) mockResponse.proxy();
    }

    public void tearDown()
    {
        mockRequest.verify();
        mockResponse.verify();
    }
        
    public void testDoGet() throws Exception
    {
        mockRequest.expectAndReturn("getParameter", "command", 
            "SELECT...");

        // Verify that the result of executing the command has been
        // stored in the HTTP request as an attribute that will be
        // passed to the JSP page.
        mockRequest.expect("setAttribute", C.args(C.eq("result"), 
            C.isA(Collection.class)));

        servlet.doGet(request, response);
    }

}

Update: Thanks to Chris Lenz, fixed the testDoGet() method and removed not needed asserts

The update process is quite nice too: we change the RDF file in CVS. Then every night Gump builds the Cactus project documentation (we transform the RDF file to HTML using XSLT in this process) and if the build is successful, upload it to http://jakarta.apache.org/cactus. Thus we get automated updates from CVS to your RSS feed reader!

I've, at last, discovered the joy of RSS syndication. This is a revolution for me. I was used to opening my browser every morning and scanning my favorite sites... No longer. I'm now using FeedReader, a very nice windows application that automatically polls your favorite RSS feeds and displays them in a nice manner.

The other nice features are

• automatic pop up notification when news arrive
• automatic scans for news every N minutes
• start automatically upon windows start up in the systray
• integrated browser (IE) for easy navigation

To get you started, here is my list of subscriptions (FeedReader format). Drop this file in your Documents and Settings/yourusername/Application Data/FeedReader directory or simply merge it with your existing subscription file.

Update:: I've just discovered that there are some other very nice RSS feed readers (I haven't tried them yet though): Syndirella and SharpReader.

Update:: Thanks to Scott Stirtling for the best RSS Feed Reader / News Aggregators Directory he's seen.

• Whenever you check out a file, the Subversion client puts a pristine copy of this file in a special .svn directory (equivalent to the CVS/ one for CVS). This means that whenever you later edit this file, Subversion is able to tell you without going to the server what changes you have made to the file. It is also able to revert your changes again without calling the server side. But the best is that when you perform a commit it is thus able to only send the difference whether you are working on text or binary files! That really rocks!
• Ability to version anything (directories, etc)
• Supports move of files, directories, branches, etc
• Uses HTTP(S) and WebDAV (well, a specific version of it see below)
• Some nice GUI clients (especially TortoiseSVN which I haven't tried but which looks very nice)
• Lots of another nice features described on the Subversion web site

On the negative side, it is still missing the following to be perfect:

• It does not support locks at all. CVS did not support locks but add the watch/edit feature which could be used (the current version of Subversion does not have this feature). Not having locks can be a pain when you have a big team working on binary files such as Word documents, Powerpint presentations, Rose .cat files, etc.
• Missing a nice Eclipse plugin (there is some work in progress but nothing visible at this point in time
• Does not really support the Generic WebDAV clients such as the WebFolder in Windows. Details about this can be found here.