Tuesday, January 29, 2008

Need to Slow Down the Blogging for a Bit

I'm going to have to slow down on the blogging for a while as I want to start to write some new articles for Red Hat Magazine. If you haven't seen the magazine yet, it's really worth a look. (And, I'm not just saying that because I've been lucky enough to have published articles in the magazine.) It's a great resource for information On Linux, open source software, and of course Red Hat and JBoss. And - it's updated daily.

http://www.redhatmagazine.com

OK. I spoke too soon. There's one other resource that I really want to highlight.

http://www.opensourcetesting.org

This is a great site for open source testing tools, ideas and discussions. All open source of course! ;-)

Sunday, January 13, 2008

The Father of the Bride Question

(A follow-up post to "Why Are the Fountains Broken?")

What question is asked by all fathers of the bride when a wedding is being planned? "How much will this cost me - can we make it cost less?"

In a recent post to this blog, I talked about the often forgotten cost of maintenance for automated tests. Let's now talk about how much this maintenance may cost, and how to limit that cost. In other words, how much of my project schedule do I have to devote to maintaining my automated tests? Let's walk through an example. Suppose you are building the first automated tests for a new product. For the sake of this example, let's assume that your test plan calls for you to create (50) automated tests. Let's also assume that you are using an existing test framework such as JUnit or TestNG.

The first test may take you up to a day to create as you will be learning the product under test. You will likely spend a good deal of time simply getting a simple test running. After this, you will probably be able to write perhaps 50% of the tests at the rate of two per day, and perhaps the other 50% at the rate of three to four a day. When you have them completed, you'll need some additional time to refactor and generally clean up the tests. Let's allocate three more days for that.

So, in summary:

the first test = 1 day
the next 24 tests = 12 days
the next 25 tests = 7 days
clean up = 3 days

For a total of 23 days = in other words, perhaps a month.

Now, let's think about maintanence. Let's assume that your new product ships quarterly updates which contain bug fixes and minor new features, and annual major releases. All of these new releases and bug fixes require that you write new automated tests, so over time, your test library grows and grows. But, how much of your time and person resources should you plan to spend on maintaining these tests?

To a large degree, the amount of investment in maintaining the tests will depend on how effective the tests are, and on how much the code actually tested by the tests changes. Let's say that 10% of the code in your project changes in each of your quarterly releases. Where will you spend your time and resources in updating your tests?

First, you will have to review all the tests so that you can determine which tests must be updated, and which new tests must be created. Then you have to design and implement the changes. Let's say that you can review ten tests per day, and that the time required to update these tests is on the scale of one half day for each of the 10% of tests.

So, in summary, maintenance on the first set of tests will cost:

review all the tests = 5 days
update the tests = 3 days

For a total of 8 days. Or, about 25% of the time to write the tests in the first place. If this sounds expensive, well it is! What we need to do is to find a way to reduce the time needed to review all the tests, because remember, the number of tests is always growing.

What's the answer? There's no silver bullet to the mundane task of maintenance. But, part of the answer is the mundane task of documenting the tests so that the specific tests that ought to be changed can be easily found.

This documentation should record not just the design of the tests, but the goals of the tests. This documentation becomes the "institutional memory" of the test automation in that it is persistent, and it outlives any one person's involvement with the project or the tests. Another part of the answer is to keep this documentation in a form that can be easily reviewed, that maps directly to the project requirements' definitions, and is easily edited. Javadoc is a great approach for this as it enables you to keep the test documentation in the actual test source files.

A few years ago, I was involved in a situation where I was dealing locating test coverage for a specific feature within a test suite of several thousand tests. Over the years that it took to develop the tests, the knowledge of the precise actions performed by any one test was lost. The only way to review the tests was to review the source code of each test. It was possible for Development and QE engineers to do this, but other project team members who had project knowledge, but not programming skills, were not able to contriute to the review.

So, why are the fountains broken? Maybe no one planned to or was able to maintain them. Why are the tests outdated? Maybe for the same reasons. But, if you plan for maintenance and make it possible for everyone on the project team can easily review the tests in a form that they can understand, then maybe you can direct your finite test automation resources at the tests that need updating. And in the process save some time and money.

Saturday, January 5, 2008

To Make Sure the Software is Functional, Don't Forget the Non-Functional Tests

When I first heard the term "non-functional test, I thought that it was a joke. I was working for a now-defunct company on a now forgotten product. The product's code was anything by fully "functional," so when I was asked about non-functional tests, it seemed more like an appropriately funny and sarcastic comment than a serious question. It may sound odd when you first hear it, but the classification of tests as "functional" or "non-functional" actually is very logical.

What are functional tests? These are the tests that you build and run to exercise (and exorcise the bugs out of) the code that supports the functions and features of the product under test. In other words, these tests tell you if the product fulfills its functional requirements. What are non-functional tests? The best description that I've seen is that these tests verify how well the product fulfills its functional requirements. A good way to think about this difference between functional and non-functional tests is that the functional tests verify the "whats" and the non-functional tests verify the "hows."

The details of functional tests for a product will be specific to that product. For examples, tests for a text editor will be different from tests for a firewall. The types of non-functional tests that apply to a product will also depend on the characteristics of the product, but will likely include types of tests such as these[1]:

* Compatibility testing
* Compliance testing
* Documentation testing
* Endurance testing
* Load testing
* Localization testing and Internationalization testing
* Performance testing
* Resilience testing
* Security testing
* Scalability testing
* Stress testing
* Usability testing
* Volume testing

So, functional tests or non-functional tests? Which types should you build and run? For a thorough test cycle, you really need both. Hmm. What about dysfunctional tests? I'll get back to talking about these in a later post.

[1] http://en.wikipedia.org/wiki/Non-functional_tests