Last year I created a build system for a very big project here at ThoughtWorks.
It's been through many iterations using Ant, Maven and finally Ruby. It's been quite a journey but the end result works quite nicely.
Why not a standard build-tool like Maven or Ant? As I said it's a huge system with a very complicated build. We started out with Ant-scripts generated by XSLT from a configuration-file. The configuration-file declaratively described the modules and their dependencies (something like a Maven-POM). You could easily edit the simple configuration-file to change things like classpaths and dependencies between modules and creating new WAR-files. This worked quite nicely initially but the system grew and the build became more and more complicated. For example, the generated build-script was several megabytes large!! The xslt transform was also absurdly large and very hard to maintain. We had similar maintainability problems with Maven. To summarize: using a standard build tool for something as big as this is simply not feasible. Build tools don't have things like data-types, control structures, object-orientation and when your build grows to become a system in itself you need all this stuff.
Why Ruby? This was basically a personal preference, any scripting language would do, like Python or even Perl. Ruby is at the core a very clean object-oriented language (unlike Python and Perl) and on top of this a lot of nice Perl-inspired syntactic sugar that makes it possible to write compact and good looking code. The perfect combination! Almost (but not quite) brings me back to my Common Lisp days (writing this).
Due to the dynamicity of Ruby we could do lots of nice things. For example we added a new construct that you could use to make ordinary methods behave like build targets, if a target had already been built the next time you built it (with the same arguments) it would simply not get executed. (Almost like a caching interceptor.) For example:
def generate
# execute code generators
end
target_method :generate
def compile
# execute javac ant-task
end
target_method :compile
def test
generate_code
# execute junit ant-task
end
build_command :test, "run unit tests"
def jar
generate_code
compile
# execute jar ant-task
end
build_command :jar, "create jar"
def all
test
jar
end
build_command :all, "default target"
So if you execute all
the targets generate
and compile
would only get execute once. Without this they would get executed twice with catastrophical effects (since test execution take almost 95% of the build time).
The build_command
modifier makes the method a target and also adds it to the list of externally callable commands (with an optional description). A little framework makes it possible to call these from the command-line or a simple console (keeping caches in memory thus speeding things up).
There's a RCR (Ruby Change Request) to make the def
method return something sensible (now it returns nil
) so you could instead do:
target_method def generate
# execute code generators
end
Like a modifier in Java. This should look much better!
Reusing Ant tasks in Ruby This project used code generation quite extensively with custom code generators and XDoclet for Hibernate configuration files. (Personally I'm not very fond of code generation, but this choice was made long before I came in.) All of this was implemented with Ant-tasks. Another reason for reusing Ant-tasks is that starting things like javac
and jar
has a high startup time but when started they run quite fast. (Unfortunately jikes
fell over on this massive codebase.) Within Ant these tasks does not have the same problem.
So reusing Ant-tasks was simply a must.
We ended up creating a small "Ant-server", a simple program that once started took Ant-snippets on it's standard input and printed out the results on the standard output. With some work on the Ruby side to manage this process correctly it all you needed to do to execute Ant-tasks within Ruby was:
ant(%{
<javac
encoding="#{default_file_encoding}"
destdir="#{destdir}"
classpath="#{to_classpath(tools_classpath)}"
debug="true"
srcdir="#{src}" />
})
(The "#{}" inserts the corresponding Ruby-expression into the String, almost like Ants "${}" but much more powerful.)
Summary The end-result was still quite a large codebase but much more maintainable. We reused the declarative configuration file that was used to generate the Ant build so we could coexist neatly with the Ant build. Editing the declarative configuration file to change things like classpathes was really simple and did not require editing the Ruby-code. We used some of the object-oriented constructs in Ruby but the next time I would probably take this a step ahead and make things even more object-oriented, so instead of "compiling" you would ask a "module" to "build itself". This should make the build code much easier to understand (we had the typical smells of too long parameter lists and so on).
I would also look into Groovy, it has very cool ways of calling Ant scripts. Since it also has some of the niceties of Ruby it could be the perfect scripting language for doing complicated builds.
(We also used a cool invention by Aslak that made it easy to parallelize and distribute the build. More on this some other time.)]]>
