Linus seems to have not chosen any of the established change-oriented SCMs (Arch, Monotone, Darcs) and have instead chosen to implement his own: Git.

Does anyone know more about this Git? Would it be suitable to use for projects other than the Linux kernel?

I can't even find any documentation for it...

While reading my usual MacOS rumour sites this morning I came on to this article about Bush and his iPod. While the article itself isn't very interesting I did get very impressed about the "Ajaxish" (sorry about the buzzword drop here, it's just the best way to describe it) way of turning pages. If your mouse hovers over the right hand side the "Next page" widget lights up, if you click the next page comes up without refreshing the whole browser. If you hover over the left side the same goes for turning to the previous page.

Very nice! Anyone know how this holds up accessibility wise?

By popular demand, I present to you the follow-up to my extensively slaughtered last blog post. This time I'll try something new; balanced analysis! ;-)

Just a disclaimer to start off, I have extensive experience designing, developing and maintaining J2EE based systems but I have no practical in-production experience with FastCGI whatsoever (apart from my own dabbing around with various internal ThoughtWorks stuff). Yeah, yeah, call me a kid that's too young to remember and blah blah. (Yes, I'm looking at you Jason.) ;-)

Without further ado, these are the various issues that I see with both solutions to the scalability problem (jumbled together with latency and perceived speed, but hopefully you'll get my point):

Sharing data between processes

Both J2EE and FastCGI share this problem; J2EE applications that needs to scale is usually deployed to a cluster with multiple processes on multiple machines and FastCGI of course is always running in multiple processes. There are two solutions to the problem, implement session affinity so that all requests in a session always ends up at the same process, or implement a way of sharing session data using some form of intra-process communication. Doing the latter is actually so easy and well-understood nowadays (both in J2EE and FastCGI) that session affinity is rarely needed (session affinity also has a availability. The main strategies is to use shared in-memory storage (memcached in Rails land), session replication (JBoss uses JavaGroups for this), storing the session state in a database (both Rails and J2EE can do this) or on the filesystem (this is the default out-of-the-box behavior of Rails). Storing session state on the filesystem is a simple solution that actually works amazingly well, distributed filesystems with good enough performance is very easy to set up and use in all environments nowadays. As load increases it's probably worth investigating the other options.

Cost of running multiple processes on the same node

Processes are more expensive than threads because of the cost of maintaining an extra heap and other kernel resources. That said, most modern operative systems have highly optimized process management, Linux can for example easily run thousands of processes on well-speced hardware and with the new scheduler in 2.6 the limit should be even higher. I'm not completely up-to-date with this but the JVM used to fall over after a couple of hundred threads. Also, the standard solution to the problem of limiting multi-threading issues is to limit the number of objects shared between threads, thus the benefits of just having one single heap is slightly limited.

Simplicity of programming models

FastCGI has a simpler programming model as each request is processed completely separated from other requests going on at the same time. This may not be a big difference to application developers but it does makes a huge difference to the framework code in Rails. I've discussed simplicity before (it is one of my favorite subjects) and it should mean that if this simplicity is harnessed a framework based on FastCGI can be easier to maintain, stabilizes faster, could get more committers and have features added to it in a more rapid pace. Of course, simplicity is an elusive thing and if the Rails developers does too many design mistakes then they might not be able to reap the rewards. Complexity always lures around the corner, ready to attack when you least suspect it. Personally I have quite a lot of confidence in the Rails developers: The introduction of Ajax to Rails was one of the cross-roads where complexity could have started to cripple the entire framework but the implementation is such a wonderful showcase of elegance and simplicity (where as my own design quite frankly... ehm... sucked...).

Multi-threading safe objects

J2EE's multi-threaded design requires shared objects to be designed and implemented with multi-threading in mind. Again, this may or may not be a problem for application developers; in Struts for example the action instances are shared by multiple threads where as in WebWork each request gets its own instance. It does make a big difference to the framework though. A lot of the objects that are shared by multiple threads are in the framework and this is tricky indeed to get exactly right. Just an example of this: We did a minor upgrade of Jetty on a system I maintained some years ago. Everything seemed to work fine but blew up completely on some machines in our server farm. Eventually we tracked it to that the version of Jetty we upgraded to didn't function properly on a multi-processor machine. Granted, our QA process was lacking as we didn't do regression testing on the same hardware as we had in our server farm but this was a small shop that just couldn't afford this.

Cost of intra-process communication between webserver and FastCGI process

This extra cost is limited if the webserver and FastCGI process are running on the same machine, most operative systems has highly optimized IPC solutions. If they are deployed to two physical node, which is the case in a FastCGI cluster, then there is certainly an extra cost which can give high latency. Most J2EE applications (particularly enterprise ones) are deployed in a double firewall architecture where the J2EE application server and the webserver resides on two different machines. In this case the J2EE solution also suffers from this problem.

One database connection allocated to each FastCGI process

For each FastCGI process you need to allocate one connection and this connection is open for the entire lifetime of the process. When using connection pools in a multi-threaded process the connection is only allocated during actual transactions with the database. This could potentially limit how far you can scale a FastCGI architecture although how high this limit is is very hard to say as it depends on the type of load your application have, what type of database and so forth. (Remember that many databases can also be clustered to handle a large amount of connections.) Also, it is best practice when using for example Hibernate to allocate a connection early in a requests lifecycle and keep it to the end, so this problem could show up in a Java application too. Also to note is that one of Rails caching strategies is to generate static HTML to the filesystem and then let the webserver deliver it, this means that a FastCGI process isn't used for that request at all. How applicable this caching strategy is depends very much on your application. On some applications (blogging engines and content management systems for example) this can be used extensively where as on other applications (internet banks for example) it cannot be used at all.

Summary

Do I dare a summary or will I get shredded to pieces in blogsphere? Well, I suppose I'll get abuse anyway so I'll just give it a go: Both FastCGI and J2EE have both been proven to scale in production through many years and are both very well understood. They should both be considered low risk. Where FastCGI has a simpler programming model J2EE seems to possibly have lower latency and a higher theoretical limit to what dimensions it will scale to (although where this limit is is hard to tell).

Both FastCGI and J2EE are good solutions to the scalability problem.

There's a lot of buzz around the Ruby on Rails framework at the moment. Something that doesn't get a lot of attention is it's scalability solution: FastCGI. Partly because it doesn't seem very exciting on first glance and partly because it's so fundamentally different to how "enterprise systems" have traditionally scaled. I think Java developers in particular should invest some time in understanding FastCGI.

A Java virtual machine is extremely expensive to start. When started it occupies huge amounts of memory and system resources. This property of the Java platform has led us down the path of scaling using one single virtual machine per physical machine.

One way of doing this is to use some form of non-blocking event-driven architecture and have a fixed small number of threads (typically one per CPU). Each thread handles a number of requests at the same time, as one request waits for IO it moves on and processes another request. This is a complicated (but interesting) way of building systems and not very well suited for non-senior developers. Most enterprise software projects have a large portion of junior or mid-level developers so this is not really a practical way of scaling the Java architecture for enterprise systems.

The other way of scaling this is with a larger number of threads, each thread processing only one request at a time. As one request waits on some form of IO operation the thread blocks and dispatches to another thread. This approach is much easier to develop in and it has been shown to if not scale as well as the non-blocking IO solution it can still scale pretty well. (Take a look at for example the mixed-threading model of JRockit, where a lot of IO optimizations are done on the virtual machine level.)

Another interesting issue is how expensive resources are handled (typically database connections). In a one-VM solution they are pre-allocated in pools so that the cost of allocating one doesn't get incurred on each request. As a request is completed it's very important to return the connection to the pool so it is available to other requests. Another complication is that the heap is shared by multiple threads and all shared objects needs to be built for multi-threading safety. Designing and implementing multi-threading safe objects that doesn't deadlock and have a high throughput is extremely hard. Because of this complexity the traditional solution has been to simply dodge the issue and minimize the amount of objects that are used by multiple threads at the same time. Those objects that are used by multiple threads have little or no state. This effectively solves the problem, but minimizes the benefits of having a shared heap. Because many systems are clustered on multiple physical nodes the shared heap can not even be used as a means of inter-process communication.

These are all interesting characteristics of the typical enterprise software architecture. We've slowly learnt to deal with them and have built up a toolbox of quite effective tools around them (J2EE being one of them).

Ruby on Rails and FastCGI scales in a completely different way.

FastCGI is an extension to the old and ultra-simple CGI architecture which to put it bluntly doesn't scale one single bit. For each request a CGI implementation will fork off a new process containing the application code with a bunch of environment variables telling the application what to do, it will capture the output of the process and send it back to the web client. Starting a new process is usually a very expensive operation (depending on operative system and what type of process and so on) and no resources can be kept alive between requests so database connections and so on will all have to be reallocated on each request.

In contrast, a FastCGI implementation will on startup pre-fork off a number of CGI processes. Each process will listen to standard input (or any other IPC solution such as named pipes, domain sockets or even network sockets in case of a cluster). As a request comes in an available process is chosen and the content of the request is sent as name-value pairs to the process. The FastCGI implementation captures the output and sends it back to the client. The process is then returned back to the pool of available processes.

This means that each process can pre-allocate one single database connection (for each database that it talks to). There are no issues of multi-threading as each process processes only one request at a time. No objects needs to be written to handle multi-threading, as there is just one single thread per process. Expensive resources doesn't need to be allocated in pools and application code doesn't need to return the resources once done with them. Complicated non-blocking IO solutions or muxer/demuxer architectures doesn't need to be used. You can even allocate FastCGI processes on multiple physical nodes, effectively implementing a cluster. In high-security situations a double-firewall security architecture can be set up so that the web-server is protected by one and the back-end FastCGI servers are protected by an additional one.

There seems to be some indications that FastCGI scales at least as well as the typical application server architecture. If this is the case then it's great news. Dealing with the complexities of a multi-thread/one-process system are very expensive. In practice the real performance and scaleability from FastCGI applications might be even higher as the much easier development model decreases the risk for programmer errors.

Java 5 seems to have done some clever optimizations on some platforms for starting up additional virtual machines (you pay up front for the first one, the rest is very cheap). Maybe it's time we try this stuff out in Java too?

For more information:

http://www.fastcgi.com/

http://www.fastcgi.com/devkit/doc/fcgi-perf.htm

http://cryp.to/publications/fastcgi/

...and much more.

Update 1: I may come off as bashing the multi-threading solution here, which really wasn't my intent. I am intrigued by the simplicity of scaling with processes and am curious as to whether this will actually work.

Update 2: Cameron Purdy points out a very valid flaw with the FastCGI architecture; if you for example create one hundred FastCGI processes per server in a 20 blade cluster then each separate process needs to allocate one database connection. This would amount to 2000 database connections which occupies a lot of memory (both in the database server and on the client) and bluntly put just doesn't scale. ;-) Does anybody that have real experience using FastCGI know how to solve this problem? Is there a solution or does FastCGI simply not scale to these dimensions? (Please ignore the unfriendly tone Cameron uses in his post, which surprises me as it's quite unlike Cameron "Peace" Purdy. It's really sad that Hani is making such an impact on the Java blogging community.)

40 comments

You may be wondering why there have been so little activity from me the latest, well, almost a year. Ok, you might not be wondering but I'm gonna tell you anyway:

I found paradise.

Let me put it this way: I currently live about 50 meters away from one of the most beautiful beaches in the world. I can get up 6.30 in the morning, grab my surf board and watch the sun rise out in the line up. On the ferry to work I pass through amazing nature and get off just in between the Sydney opera house and the bridge (if you're lucky you can even see dolphins or whales on they way). I can get my car and drive north past spectacular beach after spectacular beach. Go on weekend trips to the vineyards, the best diving in the world, or some extraordinary sailing.

Who really cares about OSS, IoC, AOP, RoR, Ajax, XP, OSX, etc anymore? Well, ok, I do... it's just kinda in the background nowadays... I want to care, but life is just too damn good down here.

(Btw, we're still looking for top people in ThoughtWorks Australia. You want to seriously increase your life quality and work with the best people in the world, drop me an email.)