Continuing with my current build-mania, I'd like to propose the idea of a distributed build architecture. I'd love to see my favorite continuous integration tools (CruiseControl, DamageControl and Continuum in the future) support this notion in the future (I know they're thinking about it already!).

So what is the need for a distributed build?

I can see several use cases:

• building on several JDKs
• building on different OS platforms
• building with different environment setups (for example building with different application servers, different browsers, different databases, etc) to validate that a product integrates well with various environment setups
• delegating the build load on several machines when the build starts to take too long (of course, the first solution should be to try to lighten your build as much as possible)

A proposed architecture

Disclaimer: this only ONE potential solution. There are lots of other solution probably even more valid than this one. Please feel free to add your ideas as comments to this post.

It could work as follows:

1. The central build machine (aka the build orchestrator) decides to start a build. The orchestrator can be one existing continuous integration tool like CruiseControl, DamageControl, etc. They can trigger a build on anything they want: time-based, change-based, manual, continuously, etc. The orchestrator sends a build request to the space. The request contains all the information about the requested build (e.g. JDK to run on, OS to run on, App.Sever/DB/etc to run on)
2. The space holds all requests. It chould be a good idea to provide a browser to see pending requests (preferably using a simple HTTP browser so that people who wish to contribute can see what type of builds are required). In any case it's important that the space be transactional (Note: I'm not sure about the word "transactional". What I mean is that a request cannot be read by several build agents at the same time)
3. Build agents listen on space build requests objects that match their capabilities. Using Jini/Javaspace would be nice here because (among other things) agents would be able to easily listen to requests with Jini attributes (OS, JDK, etc). Once they read a request they start a local build and publish the result to the space as a Result object
4. The build orchestrator listens to Results object, and generate result reports, aggregating all results. Build results could contain anything required: result of the build, logs, generated artifacts, etc. The orchestrator gets the data from the Result object and perform usual build operations (publishing, build result notification).

Of course there would be several details to sort out, like should we send 2 Requests object for each build need so that we can compare the results and only accept the result if they match, etc.

Conclusion

I think this type of distributed build could be especially interesting for open source projects in order to build an active community around a project. This would be yet another way in which people can contribute to an open source project: by lending some of their machine CPU to perform continuous integration builds of this project. This usually makes sense as open source projects may be low on hardware resources and lending some would help. Of course it also bring its challenge of security issues that would also need to be implemented...

Would you like such a distributed build system? I personally prefer this architecture over one where the orchestrator directly sends build requests to build agents as I find it more scalable and more flexible.