We seem to have a big problem in the Java persistence community, and it's one that seems to have a surprisingly simple solution.

Storm clouds

About mid-May 2004, we began to suspect that things weren't quite right, about the time the SE/EE Executive Committee voted on JSR-243 for JDO 2.0. Several EC members voted against the JSR, although it did pass. You can read about it here. [Full Disclosure from Geir : I'm the Apache EC rep and voted yes because I think that despite assertions to the contrary, there is no overlap with existing technology, and if there *is* overlap, competition is good - let the market decide.] Looking back, there was a hint of storm clouds on the horizon with Oracle's comment, namely that JDO 2.0 overlaps work being done in the EJB expert group. We didn't think anything of it at the time ("JDBC overlaps work being done in the JNDI expert group" if you believe that both are ways of retrieving data) but with the benefit of hindsight, it's almost as if Oracle was warning us about what was coming.

Looking back at the comments, we also have several references to J2EE. JDO isn't part of the J2EE spec - JSR-243 notes it will work to be more "aligned" with J2EE - and J2EE isn't the only platform under which people build applications. JDO is important for non-J2EE developers working on J2SE applications. With further hindsight, we see from those comments that several platform vendors appear to have identified a developer need unfilled in the J2EE platform.

The Three Tiers of the Persistence Inquisition

We tend to think about persistence tools in the way we think about any other set of developer tools for a given technology. Generally, once software engineers have enough experience in a problem domain, they will produce a set of development techniques or models which are designed to address different levels of complexity in that problem domain. They tend to build on one another and if lucky, there's a reasonable path for moving 'up the chain' as an application's complexity increases. This model and tool approach is true with persistence as well :

1) Simple and Uncomplicated : At the bottom of the complexity stack of Java data persistence is the old 'fast and dirty' standby that we all know and don't truly love - JDBC. We've all used it and gotten quite a bit done with it, but it's not terribly pleasant when we live in an object-oriented world (or even a data-structure world). We tend to write simple tools to help us around the "bare metal", and after a while, we stand back, look at those simple tools, and and decide that we need a better level of abstraction to handle the domain model that we work in. That leads to tools and models of medium complexity.

2) Medium complexity : This consists of tools that allow us to work easier with the everyday objects and data structures that make up our applications. This isn't a new idea, and probably extends from seeing object database systems and, not being able to actually find any in the market ;), wanting to use general purpose RDBM systems for the same purpose. There is a wide range of approaches to this problem, from populating JavaBeans from XML (and back) to POJO persistence, with solutions ranging from standard to proprietary, commercial to open-source. Examples include the Java JDO specification with it's implementations, commercial proprietary O/R systems like TopLink and the current crop of open-source solutions like Hibernate, OJB, Cheyanne, et al.

The basic idea in middle-complexity-land is fundamentally about persisting POJOs (Plain Old Java Objects). These approaches generally are very convenient to use, allow you to work without any component contracts forced on your classes by the environment, and more or less give you the freedom to easily save objects or graphs of objects to some kind of data store and get them back later. They tend to be "deliberate" - the programmer makes the decision about when to save with some kind of explicit POJOFramework.save(myObject) call or -ish.

They also are very popular (and becoming increasingly so) because most applications seem to be at this level of complexity - you just want to work with your domain objects in a deliberate way. (Or even just use to 'persisted data structures', which is my favorite use for Hibernate, for example).

3) High Complexity : To address high levels of complexity, we have complicated frameworks like EJB that provide middleware services such as transactions, security, data storage, component relationship management, etc. This third tier is generally very powerful, very complicated, and seems to be often misapplied. Because it's part of the 'enterprise' Java standard, it's the 'natural' choice for persisting data in an 'enterprise' application, although there are other solutions that, depending on the application, would be just as appropriate and easier for the developer to use, deploy and maintain - solutions from medium-complexity-land that aren't part of the J2EE spec.

J2EE, EJB and Thee

With the preceding discussion in mind, lets look at what's been happening with the next version of J2EE. According to the text of JSR-244, the theme of J2EE 1.5 is "ease of development", with the goal of addressing the needs of a wide range of developers, "including less sophisticated developers". This is a laudable goal, and quite frankly, it's about time. By adding new features that enable better tool support, ease of application development, and ease of maintenance, the Java will hopefully become more widely adopted and remain a competitive platform for enterprise development.

Expert groups that are part of the J2EE "umbrella" are working to support that goal, and this includes Enterprise Java Beans (EJB). The JSR for EJB 3, JSR-220, has the following as its declared purpose :

"The Enterprise JavaBeans architecture is a component architecture for the development and deployment of component-based business applications. The purpose of Enterprise JavaBeans (EJB) 3.0 is to improve the EJB architecture by reducing its complexity from the developer's point of view."

It makes it pretty clear that EJB is a component architecture (and happens to include the 'high complexity tier' of data persistence) and the intent of this version is to improve the architecture by reducing complexity. This is just fabulous.

POJO Persistence and Mission Creep

As we've heard at TheServerSide Symposium and other places, the EJB expert group is working on what amounts to splitting the EJB spec into two concurrent 'tracks'. They don't talk about it like this, but it's a reasonable interpretation :

- an addition of a new and incompatible POJO persistence model
- a continuation of the EJB 2 component architecture with new POJO persistence model features backported (changes to query language, for instance)

We think that one conclusion that we can safely draw from this is that the EJB3 expert group recognizes that EJB2 persistence doesn't satisfy the needs of a large number, if not the majority, of enterprise application developers. Further, they recognize that POJO persistence is a valid persistence strategy and developers like to use it for their applications. They are so interested in having it available, the EJB expert group is willing to alter the objective of the JSR to accommodate.

I think that we should appreciate the EJB3 EG for the courageous step of effectively saying "We realize there's more to enterprise data persistence than EJBs and we want to do something about it" It's not unreasonable, though, to presume that this 'bifurcation' of the EJB spec will be fairly confusing for developers because now EJBs are going to be "EJBv2 EJBs" and "persisted POJO EJBs".

Paying attention to customer demand is laudable, but when you consider that these two technologies (POJO persistence and EJBs) are arguably two distinct solutions to data persistence appropriate for differing levels of application complexity (and EJB is more than just data persistence), this bifurcation of the spec begs a question :

"Could we prevent an over-broad and potentially confusing EJB model by simply adding POJO persistence separately to the J2EE spec for 1.5?"

One would assume, given the investment made by users and vendors of J2EE, that an incremental revision of the J2EE spec (from 1.4 to 1.5) would presume a compatibility path that preserves existing investment and allows those implementations to smoothly take advantage of new features. Users aren't going to want to rewrite critical business applications just to keep up with vendors. What we seem to be seeing here is a discontinuous change from EJB2 to EJB3 and there is no smooth path from EJB2 applications to EJB3 applications.

To answer the question above, we could achieve the same result by making a POJO persistence model available in J2EE distinct from EJB without compromising EJBs, which are a valuable and appropriate solution for a small percentage of highly-complex enterprise applications. Not everyone needs EJB, but when you do, you do!

The obvious solution - "Well, you've had a JSR for POJO persistence for some years now... JDO - why not add it to J2EE?" - is an answer that no one seems to want to hear because there appears to be a war going on between the JDO and EJB communities. Technology battles aren't new, but this one is astounding in it's irony. The EJB3 group, by adding POJO persistence, is validating the whole argument for JDO or a JDO-like technology in J2EE. There will be arguments about technical details such as choice of query language, approaches to detaching object graphs, etc but at the end of the day, the EJB EG is advocating the addition of a technology approach that already exists in the "Java ecosystem" - JDO - into J2EE.

We're optimistic that this can be resolved. We don't believe that a spec lead can be held responsible for the public behavior of the expert group members or non-EG members of the community. However, as both specs are Sun sponsored, both spec leads are Sun employees and most importantly, all parties agree on the fundamental need for POJO persistence in J2EE, it's conceivable that through a concerted effort by Sun to recognize the gap in J2EE and work towards an architecturally clean solution, partisans from both sides can work with each other rather than against each other.

Solution!

The simplest solution is to add POJO persistence (via JDO 2.0 or other) to the J2EE umbrella. As a bonus, require that POJO persistence and EJB work from the same persistence layer, so that concurrently deployed applications written using both technologies have coherent views of the same data. (And if vendors are smart, they'll provide a JDBC driver to access the same persistence layer....)

While fixing J2EE to support what programmers want today, it also offers a compelling path to J2EE from J2SE, because J2SE applications that are written using the same POJO persistence model will be able to migrate to the J2EE environment without requiring changes to their persistence model. Further, we can take a lesson from Microsoft's ADO and tools such as Hibernate and OJB in Java - that there is a strong demand for persistent objects in the base platform. So lets also ensure that we take what we learn from J2EE 1.5 and push what we can 'down' to J2SE 1.6. (On that note, how many people know about the rejected JSR-20?)

JDO 2.0 just got started. Any of the supposed deficiencies in JDO can be addressed. If, in the opinion of the J2EE EG, that JDO's model is inappropriate - that recent work in open-source or commercial solutions offer compelling alternatives - start a new EG and include members from JDO as well as EJB.

We feel that this is an important problem, but a very manageable one, and hope that the broad Java community supports this approach - addressing the needs of the development community is one of the core purposes of the JCP. Lets use it.

- Geir Magnusson Jr. and Jeremy Boynes

Note: The above represents the personal opinion of Geir and Jeremy and should be not construed to represent the position of any other organization. Specifically, this does not represent the position of the Apache Software Foundation.

Full Disclosure: Geir is the Apache representative to the JCP Executive Committee and recently the current Apache representative on the J2EE 1.5 Expert Group. Jeremy is the Apache representative on the EJB3 expert group. This article is an original work by Geir and Jeremy based on public information and should not be construed to represent either EC or EG confidential information, future intention or past or current discussion. If any of this has been discussed in an EG, it is without our knowledge and we consider ourselves lucky.

Update : 08-Jun-2004 I got a nice note from Richard Monson-Haefel, a longtime member of the EJB expert group and an expert in the field. He kindly posted it as a comment. He says :

"I support Geir and Jeremy's position. While I was initially excited about a POJO persistence model for EJB, I now believe that the community is best served by having the EJB 3.0 and JDO 2.0 expert groups collaborate. These two groups should be working together for the common good of the Java community."