Stress Testing: Don't spend your life waiting for a moment that just don't come, Well, don't waste your time waiting...

I remember the situation distinctly, even though it happened over a decade ago. The product in question was a 2nd generation server. The 1st generation was successful, but like many 1.0 projects, parts of it were, well, let's just say "immature." As a result, the product was buggy and difficult (and expensive!) to maintain. The goals of the 2nd generation product were ambitious; to correct the product's design flaws, extend its feature set, and improve its overall performance and reliability.

After a long development cycle, testing started and we were dismayed to find that the re-architecting effort had completely destabilized the product. We immediately started finding memory leaks, random crashes, loss of data, and basic functions that we just plain broken. The early test results caused a large amount of angst, and also caused us to adopt a very defensive posture. We reorganized testing into discrete stages, and we determined to not advance from one stage to the next until we had established a consistent baseline of reliable product operation.

As the early stages of functional testing progressed, product instability continued, and large numbers of bugs were found, resolved, and verified. Eventually, the product's stability improved, but there serious bugs were still periodically found, so we held to our defensive posture and did not progress from functional to system-wide or stress/performance tests until functional testing was complete. When we finally determined that the product was stable enough for stress testing, the project schedule had been extended multiple times. When stress testing started, began finding still more bugs related to product's inability to function under a load over time or to handle high levels of traffic. Once again, the project schedule had to be extended as large numbers of bugs were found, resolved, and verified.

What went wrong on this project? Well, quite a bit. The planning was overly optimistic, the design was not properly reviewed or understood, unit testing was inadequate, at one point, we even discovered that if every port on the server was physically cabled and the system was not properly rack-mounted, it would tip over and fall on the floor. In addition, by being too conservative in our test planning, and too rigid in our first reaction to the system's initial poor performance, we may have delayed the discovery of some serious problems.

What should we have done differently? We should have started some stress testing as early as could find situations and product configurations that would support it. The first stress tests could have taken the form of low levels of traffic running over an extended period of time, and not necessarily a massive system load at the start. By finding tests that the product could support, we could have established a baseline of performance to measure against as the product's many bugs were resolved (and its code was changed). By waiting for "perfect" conditions that actually arrived very late in the product's development, we delayed tests that could have been run under "good" conditions.

There's a saying that, "if you see blue sky, go for it."[1] It can be like that in software testing too. If you wait for perfect conditions to run some tests, you may end up waiting for a long time, and that waiting may result in bugs being found at a late and probably inconvenient time. When should you try stress testing a product? As early as you can. Performance related problems can be difficult and time consuming to resolve. The sooner you find them, the better. So, if you see some blue sky among the bugs early in a project test cycle, go for it...

(Special thanks to Pavel!...and Mr. Springsteen...)

References:

[1] http://www.pbs.org/wgbh/amex/kennedys/filmmore/pt.html