The Father of the Bride Question

(A follow-up post to "Why Are the Fountains Broken?")

What question is asked by all fathers of the bride when a wedding is being planned? "How much will this cost me - can we make it cost less?"

In a recent post to this blog, I talked about the often forgotten cost of maintenance for automated tests. Let's now talk about how much this maintenance may cost, and how to limit that cost. In other words, how much of my project schedule do I have to devote to maintaining my automated tests? Let's walk through an example. Suppose you are building the first automated tests for a new product. For the sake of this example, let's assume that your test plan calls for you to create (50) automated tests. Let's also assume that you are using an existing test framework such as JUnit or TestNG.

The first test may take you up to a day to create as you will be learning the product under test. You will likely spend a good deal of time simply getting a simple test running. After this, you will probably be able to write perhaps 50% of the tests at the rate of two per day, and perhaps the other 50% at the rate of three to four a day. When you have them completed, you'll need some additional time to refactor and generally clean up the tests. Let's allocate three more days for that.

So, in summary:

the first test = 1 day
the next 24 tests = 12 days
the next 25 tests = 7 days
clean up = 3 days

For a total of 23 days = in other words, perhaps a month.

Now, let's think about maintanence. Let's assume that your new product ships quarterly updates which contain bug fixes and minor new features, and annual major releases. All of these new releases and bug fixes require that you write new automated tests, so over time, your test library grows and grows. But, how much of your time and person resources should you plan to spend on maintaining these tests?

To a large degree, the amount of investment in maintaining the tests will depend on how effective the tests are, and on how much the code actually tested by the tests changes. Let's say that 10% of the code in your project changes in each of your quarterly releases. Where will you spend your time and resources in updating your tests?

First, you will have to review all the tests so that you can determine which tests must be updated, and which new tests must be created. Then you have to design and implement the changes. Let's say that you can review ten tests per day, and that the time required to update these tests is on the scale of one half day for each of the 10% of tests.

So, in summary, maintenance on the first set of tests will cost:

review all the tests = 5 days
update the tests = 3 days

For a total of 8 days. Or, about 25% of the time to write the tests in the first place. If this sounds expensive, well it is! What we need to do is to find a way to reduce the time needed to review all the tests, because remember, the number of tests is always growing.

What's the answer? There's no silver bullet to the mundane task of maintenance. But, part of the answer is the mundane task of documenting the tests so that the specific tests that ought to be changed can be easily found.

This documentation should record not just the design of the tests, but the goals of the tests. This documentation becomes the "institutional memory" of the test automation in that it is persistent, and it outlives any one person's involvement with the project or the tests. Another part of the answer is to keep this documentation in a form that can be easily reviewed, that maps directly to the project requirements' definitions, and is easily edited. Javadoc is a great approach for this as it enables you to keep the test documentation in the actual test source files.

A few years ago, I was involved in a situation where I was dealing locating test coverage for a specific feature within a test suite of several thousand tests. Over the years that it took to develop the tests, the knowledge of the precise actions performed by any one test was lost. The only way to review the tests was to review the source code of each test. It was possible for Development and QE engineers to do this, but other project team members who had project knowledge, but not programming skills, were not able to contriute to the review.

So, why are the fountains broken? Maybe no one planned to or was able to maintain them. Why are the tests outdated? Maybe for the same reasons. But, if you plan for maintenance and make it possible for everyone on the project team can easily review the tests in a form that they can understand, then maybe you can direct your finite test automation resources at the tests that need updating. And in the process save some time and money.