The Father of the Bride Question

(A follow-up post to "Why Are the Fountains Broken?")

What question is asked by all fathers of the bride when a wedding is being planned? "How much will this cost me - can we make it cost less?"

In a recent post to this blog, I talked about the often forgotten cost of maintenance for automated tests. Let's now talk about how much this maintenance may cost, and how to limit that cost. In other words, how much of my project schedule do I have to devote to maintaining my automated tests? Let's walk through an example. Suppose you are building the first automated tests for a new product. For the sake of this example, let's assume that your test plan calls for you to create (50) automated tests. Let's also assume that you are using an existing test framework such as JUnit or TestNG.

The first test may take you up to a day to create as you will be learning the product under test. You will likely spend a good deal of time simply getting a simple test running. After this, you will probably be able to write perhaps 50% of the tests at the rate of two per day, and perhaps the other 50% at the rate of three to four a day. When you have them completed, you'll need some additional time to refactor and generally clean up the tests. Let's allocate three more days for that.

So, in summary:

the first test = 1 day
the next 24 tests = 12 days
the next 25 tests = 7 days
clean up = 3 days

For a total of 23 days = in other words, perhaps a month.

Now, let's think about maintanence. Let's assume that your new product ships quarterly updates which contain bug fixes and minor new features, and annual major releases. All of these new releases and bug fixes require that you write new automated tests, so over time, your test library grows and grows. But, how much of your time and person resources should you plan to spend on maintaining these tests?

To a large degree, the amount of investment in maintaining the tests will depend on how effective the tests are, and on how much the code actually tested by the tests changes. Let's say that 10% of the code in your project changes in each of your quarterly releases. Where will you spend your time and resources in updating your tests?

First, you will have to review all the tests so that you can determine which tests must be updated, and which new tests must be created. Then you have to design and implement the changes. Let's say that you can review ten tests per day, and that the time required to update these tests is on the scale of one half day for each of the 10% of tests.

So, in summary, maintenance on the first set of tests will cost:

review all the tests = 5 days
update the tests = 3 days

For a total of 8 days. Or, about 25% of the time to write the tests in the first place. If this sounds expensive, well it is! What we need to do is to find a way to reduce the time needed to review all the tests, because remember, the number of tests is always growing.

What's the answer? There's no silver bullet to the mundane task of maintenance. But, part of the answer is the mundane task of documenting the tests so that the specific tests that ought to be changed can be easily found.

This documentation should record not just the design of the tests, but the goals of the tests. This documentation becomes the "institutional memory" of the test automation in that it is persistent, and it outlives any one person's involvement with the project or the tests. Another part of the answer is to keep this documentation in a form that can be easily reviewed, that maps directly to the project requirements' definitions, and is easily edited. Javadoc is a great approach for this as it enables you to keep the test documentation in the actual test source files.

A few years ago, I was involved in a situation where I was dealing locating test coverage for a specific feature within a test suite of several thousand tests. Over the years that it took to develop the tests, the knowledge of the precise actions performed by any one test was lost. The only way to review the tests was to review the source code of each test. It was possible for Development and QE engineers to do this, but other project team members who had project knowledge, but not programming skills, were not able to contriute to the review.

So, why are the fountains broken? Maybe no one planned to or was able to maintain them. Why are the tests outdated? Maybe for the same reasons. But, if you plan for maintenance and make it possible for everyone on the project team can easily review the tests in a form that they can understand, then maybe you can direct your finite test automation resources at the tests that need updating. And in the process save some time and money.

To Make Sure the Software is Functional, Don't Forget the Non-Functional Tests

When I first heard the term "non-functional test, I thought that it was a joke. I was working for a now-defunct company on a now forgotten product. The product's code was anything by fully "functional," so when I was asked about non-functional tests, it seemed more like an appropriately funny and sarcastic comment than a serious question. It may sound odd when you first hear it, but the classification of tests as "functional" or "non-functional" actually is very logical.

What are functional tests? These are the tests that you build and run to exercise (and exorcise the bugs out of) the code that supports the functions and features of the product under test. In other words, these tests tell you if the product fulfills its functional requirements. What are non-functional tests? The best description that I've seen is that these tests verify how well the product fulfills its functional requirements. A good way to think about this difference between functional and non-functional tests is that the functional tests verify the "whats" and the non-functional tests verify the "hows."

The details of functional tests for a product will be specific to that product. For examples, tests for a text editor will be different from tests for a firewall. The types of non-functional tests that apply to a product will also depend on the characteristics of the product, but will likely include types of tests such as these[1]:

* Compatibility testing
* Compliance testing
* Documentation testing
* Endurance testing
* Load testing
* Localization testing and Internationalization testing
* Performance testing
* Resilience testing
* Security testing
* Scalability testing
* Stress testing
* Usability testing
* Volume testing

So, functional tests or non-functional tests? Which types should you build and run? For a thorough test cycle, you really need both. Hmm. What about dysfunctional tests? I'll get back to talking about these in a later post.

[1] http://en.wikipedia.org/wiki/Non-functional_tests

That time of year again!

Frohe Weinachten! Boun Natale! Joyeux Noel! Merry Christmas!

(I'll blog about software testing after the holidays!)

Here are some scenes from snowy Boston...

When Less May Be More - A Lighter Weight Test Plan

I was talking to a couple of colleagues about test plans this week; both the format of the plans and the content.

The IEEE standard is well, a "standard." It may, however, be intimidating to people unaccustomed to software testing. And, it lacks an easy to use construct for differentiating between classes of tests. The discussion of test plans came up this week as one of my colleagues is trying to - gently, but effectively - introduce a structured test process into an organization whose members are largely unfamiliar with it. He wanted very much to take a medical approach to this and "first, do no harm." For him, a lightweight adaptation of the the IEEE standard is a better starting point. We came up with this template for a test plan:

Introduction - what are we doing?
Test Strategy - how are we doing it
Test Priorities - what's most important
Scope - What's tested?
Scope - What's beyond the scope of testing?
Test Pass/Fail Criteria - how do we know that it's good or bad
Test Deliverables - docs and programs that we'll build
Test Cases - Functional Tests
* Tests mapped to product features
Test Cases - Non-functional tests - whichever apply[1]
* Compatibility testing
* Compliance testing
* Documentation testing
* Endurance testing
* Load testing
* Localization testing and Internationalization testing
* Performance testing
* Resilience testing
* Security testing
* Scalability testing
* Stress testing
* Usability testing
* Volume testing
Test Environment/Configurations
Responsibilities - who's doing what
Schedule/Milestones - when are they doing it
Risks and Contingencies - what might go wrong and how we'll handle it
Approvals - do we agree?
References - pointers to background docs
Revision history - why did the plan change and how?
Appendices - anything else?

It's worth noting how this template explicitly separates the descriptions of functional and non-functional tests as the differences between these test class definitions may be new concepts to some of the team members. What's that? Are these test class differences also a new concept to you? I'll discuss this subject in the next post to this blog!

Ref:

[1] http://en.wikipedia.org/wiki/Non-functional_tests

P.S. My Fedora friends have picked up on this test plan outline - it's the basis for their test plan template: https://fedoraproject.org/wiki/QA:Test_Plan_Template

Why Are the Fountains Always Broken? (The forgotten cost of test automation: maintenance)

I'm writing this post from just outside of Boston. By American standards, Boston is an old city. One of the distinguishing features of the city is its collection of public parks. These parks are dotted with fountains which vary in ages and design from classical and historic to modern and avent-garde. These fountains, however, all share one common characteristic.

They always seem to be broken.

Why is this the case? It is certainly at least partially due to the ravages of harsh New England climate (As I'm writing this, the Boston area is having its second snow and ice storm of the young 2007-2008 winter) on outdoor plumbing. There is, however, I think another possible reason.

When we (we human beings, that is) build something, we often forget about the cost of maintaining it after it is built.

In the case of the fountains, the lack of maintenance may be caused by the fact that in any year's budget, other budget responsibilities such as health care, roads and bridges, and public safety are higher priorities. While it may be possible to attact funding for new construction of exciting or ground breaking public places, maintenance is, in contrast, considered a mundane exercise.

The same pattern can be seen in software enginering. For example, in the development and maintenance of automated software tests.

In the course of developing a plan for automating tests, we consider the investment in time and resources to design, build and debug the tests. Once written, however, the tests will lose their value unless they are maintained and kept in synch with the software project that they are intended to test. As the project procedes from release to release, new features are added, and new tests are needed. But, the existing library of tests may either begin to fail due their falling out of synch with the project code. Worse yet, the existing tests may provide a false sense of security as they may continue to run without error, but fail to actively exercise changed, and therefore, at risk, areas of code.

Automated tests are essential to any software test effort. The tests will pay back the investment you had to make to build them many times over during the life of a project. But, don't think that the tests are "done" just because you've completed building the first version of them. You will have to revisit and review and update the tests to keep them in working order. So, when you build new tests, don't forget to plan for maintenance. Part of this planning involves good test design, so that the tests can be modified as needed. Another part of this planning involves remembering that this maintenance will require time and human effort.

Why are the fountains broken? Maybe because no one planned to or was able to maintain them. Are your automated tests running correctly today? Even if it has been several months and project updates since they were first written or last reviewed? It might be a good time to look for leaks...

Why Doesn't This Work? It's Broken. That's Why!

Here's a rathole that software test engineers often fall into. When you test software, you frequently encounter situations where some bugs block certain sets or types of tests. Until these bugs are resolved, you often resort to "workarounds" to avoid these problems and to enable you to continue to make testing progress. The danger is that in the course of "working around" these problems, you forget that the workarounds each have at their core an actual bug. And, if these bugs are not correctly resolved, you may never catch them. But, if you don't, your customers probably will.

So, in your earnest attempt to make as much testing progress as you can, even with immature and buggy software, don't forget to remove any workarounds that you put into place. Each workaround should always be treated as a "canary in a coal mine" and as a pointer to a potential serious bug. A good way to deal with these workarounds is to track them in your bug tracking system along with the bugs. You should close out one of these workarounds when the underlying bug is resolved and you can begin to take the road that you wanted to take in the first place! (And that will really make all the difference. My apologies to Robert Frost. ;-)

The Opposite of Fundamentals, the "Ratholes"

There's a great line from Robert Frost's poem "The Road Not Taken"[1] that I've always thought applied to software engineering and software testing. The line from the poem is: "Two roads diverged in a wood, and I took the one less traveled by, And that has made all the difference."

Sometimes however, the road not taken is not taken for a very good reason! It can be the case that this road can lead down a rathole and is based on bad software design, poor implementation, or tactical mistakes.

As a similar construct to the fundamentals (the "rules") of software testing that I first mentioned in an earlier post, I'm also going to start a series of periodic posts on mistakes that people often make. In other words, the ratholes into which we all fall when we choose the wrong road.

Ref:

[1] http://www.bartleby.com/119/1.html