Dion is writing about where to put config files.

I always try to make it possible to do as much work as possible without leaving the IDE. For example, I have something I call "standalone mode" where I create a simple StandaloneServer class with a simple main method which uses the Jetty API to start up the server directly inside the IDE. For this reason I always keep my config files around in such a way so that "standalone mode" works without requiring the use of a build script. That is I "Mimic The Deployment Structure". In this mode I try to stub out high-end J2EE services (not available in plain vanilla Jetty) such as JTA, JNDI, JAAS and so forth. Using Spring Framework this is easily done. I also try to stub out backend services that we don't have reliable connectivity to.

The released app is usually deployed to a real appserver where JTA would take care of committing transactions and JNDI would be used to look up pooled DataSources. Therefor I have a build script that converts the standalone mode deployment structure into a deployed production mode ear file, tweaking config files and so forth to make it behave well in a real appserver.

Doing it this way you can have the best of both worlds. You get high productivity in development because you don't need to do a full compile-build-redeploy-cycle every time you've done a change (restarting Jetty takes about 3 seconds), in fact you should never need to leave the IDE to execute build scripts during normal development. You also get an app that is a good J2EE citizen, demarcating transactions with JTA, using JNDI for DataSources and so forth. It really does wonders for productivity!

I love automatic functional tests. They are the most valuable type of automated tests you can do. They give me confidence to do refactorings without regressions. They give the business confidence to release the software when they feel enough functionality is there. They make it possible to release updates to production systems without having to do a long and expensive manual test cycle.

In short, it allows me to be agile.

So functional tests are the most valuable tests you can have. But they are also the most expensive. They are expensive to write, they are expensive to maintain, they are slow to run. When they fail they are very hard to debug and figure out how to fix them. When you change functionality you need to go through your test suite and determine what tests needs to be changed and what tests needs to be removed. As the system grows this becomes more and more tedious and hard work.

Something I'm struggling very hard with now is how to manage test data for functional tests. In order to write a simple test for, for example, a paging list I need to set up all these different objects in the domain to the right state. When some constraints or rules are changed in the domain I need to go through all this test data setup code to make everything obey the new situation.

As the system grows the problem becomes bigger and bigger. It is very hard to get reuse of the test data setup between tests as each test has special needs on the data it uses. I can't use something like mock objects to just mock out where the data is read in because this is a functional tests and I want to test as much as possible with each test. The more I test, the more valuable the test is.

I can use something like the Object Mother pattern but it only provides for individual objects. I need something higher level than this, something that can set up scenarios of objects that can be reused between tests and still allow the tests to tune these scenarios to fit their special needs.

I just don't know how to solve this, it seems to be a significant non-trivial problem of very large functional test suites.

What have other people done out there? What has worked (that I can try) and what has not worked (that I don't have to try)? I could use some help!