There used to be an SCM product, I think named Aegis, that was supposed to be another solution to this problem. If I remember correctly, you could define conditions (like regression tests) that had to pass before the repository was updated. I never used it, so I have no idea how well it worked.

That would be better. Forcing a run of all unit tests before a check-in is better than reacting later to a broken build. Unit tests should run in less than a minute with sufficient use of mock objects. Of course acceptance/functional tests could still break, but it's less likely if unit tests are at 100%.

How about properly training your staff on software development, and increasing team communication? That would be much simpler, and solve the problem without creating a big fucking mess of a build system like you have proposed here. This sort of thing treats developers like idiots. Either your developers ARE idiots (in which case, it's your own dumb fault for hiring them/working there), or they are capable of learning proper development techniques.

My big concern is the "Doesn't it take too long to build" one. I agree with your assessment that 5-10 minutes is a reasonable maximum. But builds on reasonable sized projects take much longer. Do we really have to solve the (much harder!) problem of incremental builds (ie, re-compiling only those things that depend on the changed files -- even going as far as re-running only those tests that touch changed code) to implement unbreakable builds?

Hi Michael,

I should probably have explained more this part. I am also working on a big project and our build takes at least 2 hours to build. But that's the *full* build. Our project is divided in smaller projects and each subproject can be built standalone. In order to do this we are using binary dependencies of the other dependent projects. The main continuous build is in charge of continuously producing fresh binary dependencies. A single project build should not take more than 5-10 minutes.

In any case, it should not take longer than what the developer is currently building on his local machine (it's the same build!). That said, it would be great if we could also build directly dependent projects as this would prevent non-planned API breakages. I'm still unsure how far we could go with this idea as it may take longer. Maybe randomly choosing a single directly dependent project would do the trick (even if not perfect and not guaranteeing the "unbreakable" part).

Very intresting to read indeed. But i think it is mainly suitable for a large project, with a cons of having long build times.

Peter Miller's Aegis system (which has been around since ~1992 I think) has the notion of a forced pre-commit hook. The intent/hope is that its "hooked up" to an automatic build+smoke test that ensures that the codeline builds *and* passes regression testing before committing.

Other systems that are language-aware have attempted to ensure consistency + integrity by doing semantic analysis in an attempt to guarantee that a checkin wont break a build (but without actually doing the build - since it may be prohibitively long for a large system).

These days, its fairly standard to see an shop use a pre-commit hook to ensure that some kind of build+smoke and regression testing have been run (tho possibly incremental rather than full).

What you are proposing seems to be an deployment architecture for enforcing and automating such a protocol, and grappling with issues of codebase size and build-time and distribution.

Another possibility ... rather than try to be "perfect" up front and take such pain to ensure there will be no build failures ... what if we could let the build failures happen (albeit with a culture that strongly discourages that, rather than handcuffs and a guaranteed extra overhead) AND what if we had a way of detecting *AND* correcting the build failures REALLY REALLY QUICKLY!

Suppose we attempted this kind of self-healing (autonomic) build-system. Thus when the frequent+regular integration build took place (be it daily/nightly, or more often), then only in the case of failure, there was an application with some "smarts" built-in to it that could attempt to analyze the result, and attempt to diagnose and then remedy (or at least do this to the greatest extent possible, and give that "useful" information in a timely manner via an automated notification)?

Interesting design. I can see two non-trivial problems with it, first of all: it's extremely hard to get a build verification process down to 5-10 minutes for larger projects, especially if it's going to be effective in finding problems as well. I think the key is a high level of modularisation and good dependency management, but that is something that is very hard to introduce to an existing codebase.

The other problem is also interesting: on a larger project (40+ developers) a lot of commits are going on at the same time. If the build bots runs the build verification process for each changeset serially then it may never catch up with the changesets as developers commit them. The build system needs some way of merging multiple changesets into one changeset and then run the build on them all together.

This leads me to one of my current favourite research areas: Change set oriented version control systems. Most of our current popular version control systems are file or repository based (such as CVS, Perforce, Subversion), where the repository is simply a set of directories and files, each versioned object has a set of versions. A change oriented version control system is organized along a different axis, a repository is a sequence of changes or changesets, the current state of the repository can be determined by applying all the changes in the correct sequence. Version control systems that operate in this way are for example Arch (http://www.gnu.org/software/gnu-arch/) and BitKeeper. Subversion also has something similar crafted on top of it's file/version oriented repository.

Why is this related? With a change oriented version control system a developer produce a "change" instead of committing directly to the repository. A change is a file like anything else. This change file is then sent in some way or the other to the central repository (the sequence of "approved" changes), through email, the file system or something like that. To implement the above design one could imagine a developer creating a change file and placing it in a directory, the build system polls the directory and picks up the changes, applies it to the current state of the repository, runs the build verification process and if successful adds the change to the approved sequence of changes (the central repository). In the meantime other developers have submitted new change files to the directory, the build system picks them up and so on. In case of a failure all the changes are rejected and the developers are notified via email or similar.

Maybe I've worked at the wrong places, but all my commercial experience has been with source control systems that don't allow multiple updates to the same file. A developer has to explicitly lock a file before they can check in changes (so if someone else has worked on the same file, they have to merge their changes locally). With this kind of source control system, the proposed suggestion sounds very workable (and works around some of the objections raised)

I'm new to continuous integration and only now studying CruiseControl. Are there already existing tools that do the system you propose? My main problem would be finding someone on the team with the time to set up such a system.

Aegis is exactly what you are proposing but it is not very successful in practice.

Have you heard about optimistic locking?

It is a database technique in which you never lock anything, but version it. Versioning a row means generating a new version of the row, while other transactions can still see the old version. The good thing is that databases get a lot faster, because all readers see the old data, while writers simply write the new versions. When 2 processes have conflicts, one succeeds, the other fails, rollsback and then re do the transaction immediatly.

The same happen with CVS and continuous integration, but at the source level ;-)

What you are proposing is going backwards at least 10 years... I can mention at least 10 reasons why it wouldn't work. Build breaks are not a big problem, at least not for companies that do continuous integration using available tools like LuntBuild and CruiseControl.

If it ain't broke, don't fix it.

Jon, Subversion is a tree based system, so the changeset paradigm can exist well enough there; certainly the file paradigm of CVS/RCS, makes no sense under Subversion.

As for the idea of unbreakable builds. I think I prefer applying the policy that the mainline can be unstable. The idea then is to capture as many stable builds off the trunk as possible. To avoid dealing with multiple writers you capture stable builds asynchronously by running tests, ie via cruise or damagecontrol. from stable builds you can determine stable releases. Brad Appleton could probably say more about codeline policies, but I think the basic CI approach plus policies is sufficient here. The main upsides to that approach are reduced stress on people checking in and avoiding code freezes. The unbreakable build approach strikes me as micro-freezing the repository.

The unbreakable approach strikes me not as having a scale problem, as Cedric seems to to think, but as having a social one - not enough slack.

To: extremeprogramming@yahoogroups.com
Subject: A fast, rapidly repairing, only breaks rarely build. Was "Unbreakable builds"

On Thu, 30 Dec 2004, bernard_notarianni wrote:

> I just read the new post on Vincent Massol's blog.
>
> http://blogs.codehaus.org/people/vmassol/archives/000937_unbreakable_builds.html
>
> Does anybody have seen an implementation of a similar solution?
> What were the products used? What were the issues?

I have been running a working variant of this for over a year....

The environment is embedded C hosted under Linux running under eCos on
the synthetic and real target. CVS is the SCM tool.

I am building 6 variants of the same product off a shared code base.

The full compile, splint, link run tests of all variants cycle takes
over two hours on a Pentium 4 2.4 Ghz machine.

Clearly that is an unacceptable burden on the 20+ developers.

Thus our development process is thus....

Prior to commit to the SCM the developers run a "build --for-checkin"
which compiles, links for two variants on the synthetic and real
target, and runs all unit tests on the synthetic (desktop) target.

If they add the "--ccache" option my build script uses
http://ccache.samba.org to return cached compiles, else calls in the
compiler.

If they use the "--distcc" option, my build script uses
http://distcc.samba.org/ to distribute the compile across all
available developer machines.

I do not require them to "splint" http://www.splint.org/ as part of
the precheck in build as it takes two thirds of the total
(unaccelerated) compile/splint/link/test) time.

The idea is to balance stability of the main line versus developer
time cost.

The project has a http://www.usemod.com/cgi-bin/wiki.pl wiki, one page of
which is designated the "CheckInList", where people can queue for
checkins. Once they rise to the top of the list, they have write access
to the mainline.

They are expected to tag the mainline with tag of the form
SOMETHING_MEANINGFUL_N_INITIALS_precheckin.

They can then catch up to the mainline, resolve conflicts, do "build
--for-checkin", perform a smoketest on the target and commit their
changes.

At some point, which point is debatable, they are expected to
get a second person to review their changes, if they are pair
programming, that requirement falls away.

They then tag the mainline with
SOMETHING_MEANINGFUL_N_INITIALS_checkin and add that tag to the
CheckedInList wiki page with a short description, particularly noting
any nasty surprises like database version changes etc.

They then edit the CheckInList wiki page, remove themselves and shift
the next guy to the top and give him a shout to go ahead.

A background ruby script on my machine polls CVS every 2 minutes
looking for tags of the form BLAH_checkin.

On seeing one, it sleeps 10 seconds and then pulls out a complete copy
of the source at that tag, and compiles, splints, links and tests all
variants for all platforms. It uses ccache and distcc to accelerate
the process leaving "splint" as the main time consumer. (I could use
checkin hooks instead of polling, but many other projects use the same
repository)

The result is posted on the project web site (a few lines of ruby
using the 'cgi' module.)

If it fails, the script emails me and I gently lean on the guilty
party. Usually the correction is submitted as part of the next
checkin.

Developers just wishing to catch up with the current state, rather
than checking in, are encouraged to update to a tag that has been
showned to have built/splinted /linked and run successfully.

An important design choice made early was to drop the use of
Make/ANT/scons/cook and use Ruby instead. http://www.ruby-lang.org

Experience on creating/using previous build systems had shown me that
in a mature project, the portion given to you for free by "make"
versus the code in sundry "glue scripts" becomes small. What is more
the Makefiles end up as a ghastly tangle of shell scripting, awk, sed, and
their sundry quoting conventions.

Thus I elected to choose the best of breed scripting language, Ruby,
and write two small reusable modules to do the part that
"Make/ANT/scons/cook" would give me and then be able to write the glue
in a decent glue language. The "tsort" module that is part of standard
ruby distribution did the hard part for free.

This turned out to be an excellent choice enabling a richer, easier
faster more maintainable build environment than I, or any other of the
developers on the project have ever seen.

I think it is a good idea. The quicker the feedback loop the better.

*On projects that build fast, it would be better to block the developer since any work she gets done will be a waste if the checkin fails.

* Naturally running all the unit tests after the build is a great idea too.

Just getting the basics above to work would be a big step forward. Then the next step would be to address build scalability as you mentioned.
Cruise Control is getting there (http://cruisecontrol.sourceforge.net/). Today it isn't distributed, but there is a submission in the works from a guy at SolutionIQ (Jeff Ramsdale) whom I work with, he is submitting the feature today, at that point it is up to the keepers of the source.

Discussion of this patch is at :
http://jira.public.thoughtworks.org/browse/CC-137

The notion of "Unbreakable" seems to me kind of similar to the Mozilla tinderbox procedures (http://www.mozilla.org/hacking/working-with-seamonkey.html).

I think some automation along these lines can do the trick without too much overhead. Here is how one can do it:

1. Break the work into commit/build/test[/fix] units and serialize them -- only one person commits at a time. Organize "waiting list" for committers.
2. Remember the point in time (revision number, tag, date) of the last "good" commit. Provide an easy way for developers to update to the last-known-good revision by default.

Kind of like "80% of result at 20% of the price" solution. I it can be done fairly easily with cooperating pre-commit hooks and continious build process and some lightweight scripting on the client-side. Some of it might be available already?

Hi Vincent,

Why do u need developer work-station to build on? It should not do build on developers machine to save time no?

Thanks

Besides the obvious technical difficulties I think that this mechanism is, in every which way you impose it, fundamentally bypassing the real goals of continuous integration.

The goal of continuous integration is not to have as many succesful builds as possible but to raise your team's quality awareness and to detect bugs as early as possible.

So a failing build has a function: it makes people aware of their mistakes right away and it allows you to detect problems asap.

The way to avoid failing builds is to put up a build monitor so that everyone can see right away when a build fails and to have a big lollypop to pass around. That, and indeed a person that can evangelize the importance of working precise and accurately and the importance of quality.

Making a build unbreakable does exactly the opposite: people will always have a safety net to catch their errors and quality awareness will go down. And if you lose quality awareness, you'll lose it on other things as well, not only code quality.

I was reading ur book JUnit in Action, in tat u wrote the first test case in Chapter 1, howerver u did notmhave a import statement for calculator, so u did not need a calcualtor calss and u first worte the test case and then when u get red, u build a class and then run the test case again, so this time it is green now. Is this the ideology of JUnit. To me it seems redundant

Calavista's devEDGE product (http://www.calavista.com) has had guaranteed unbreakable builds for several years now. We have been using it and have notice a tremendous improvement in productivity due to this and other features.

The broken builds kill you when you have development teams working around the clock.

We use Parabuild (http//www.viewtier.com) It provides stable aka unbreakable daily and nightly builds for us.

TT

Clearcase SCM (UCM) is often used for precisely this reason (i.e unbreakable builds).

It has the concept of a Stream for each developer.

A developer can check out and check in from his stream in isolation (other developers don't see the changes until they are delivered to a central integration stream). This has the advantage that developers changes are 'backed up' if he chooses to check in to his own stream.

The central integration stream holds the unbroken set of code (i.e. only 'working changes' get delivered)

Periodically developers rebase (get latest changes from the approved integration stream), these get merged into their own stream. They can then deliver a set of changes to the Intgegration Stream. An integration PC would perform a full build to approve the changes before completing the delivery. This makes the set of changes publicly available.

So it is like have a two-level SCM system.

The downside (I think) is it can be much slower to get someone elses changes, and slower to deliver (release your changes). Often only one developer can deliver to the integration stream at once (incurs a full build during delivery), leading to a queue of developers wishing to deliver.

I looked at how to integrate CruiseControl into Clearcase UCM, as quite a few companies have done this.

To do this I believe you have to let multiple developers deliver to the integration (approved) stream at once. Then CC kicks in and does build as normal. So this is rather like using CC with a normal single level SCM like CVS. i.e. bad changes can get into the SCM and break the build. Yes they are detected quickly using CC but they can still happen.

So if CC is used with Clearcase the integration stream has less advantages. There is still an advantage of each developer working in their own stream (i.e recovery facilities, can work on multiple machines and still see there set of in progress changes)

The idea of unbreakable builds is a good one.
reasons are numerous, but my personal favorite is the elimination of risk when submitting code. in your scenario if the 'submission" breaks, it is only a problem for the submitter. The rest of the team continues. This means as a developer, you can check in code and go to lunch. No one looking for your head if the submission fails. Microsoft implemented this codenamed "Gauntlet" for internal development. I coded a primitive version of this using VSS and a promotional model.