In Praise of Exploratory Testing

Have you ever found a bug because, in response to some unexpected behavior on the part of the software under test, you starting "exploring" and tried some unplanned tests? Were your actions guided by your past experience in testing in that you had an intuition that there was a bug lurking in the code? When you found the bug did you feel guilty because the test that you performed was not part of the test plan? Did someone complain to you that "Hey - that test isn't in the plan!"

Well, you shouldn't have felt guilty. You were actually performing exploratory testing[1].

In an earlier post to this blog, I talked about how, in order to be effective, test plans have to be dynamic, not static, and have to adapt to changes, especially with regard to risks, in a project's development. But - it's impossible to anticipate in advance every test that will be necessary. Once you get your hands on the software to be tested, you often learn more about the product.

In exploratory testing, you combine your learning about the software under test and your test design with your test execution, all at the same time, as a single action and not as separate tasks. But wait, isn't this just ad hoc testing? The question about whether there is a real difference between ad hoc and exploratory testing is frequently raised. The best description[2][3] that I've seen about the substantive differences between these two types of testing is that while ad hoc testing tends to be random in nature, in exploratory testing, you rely on your testing experience to select paths that will uncover bugs, based on the behavior of the software under test. It's very "situational" in that the tests that you create are based on the multi-step situations that you create and the situations the software under test presents to you.

I once worked for a manager who practiced ad hoc testing in an extreme form. He frequently parachuted into projects, attempted a few tasks with the software under test, reported finding numerous bugs, and then walked away. However, the bugs that he reported tended to fall into two categories: legitimate bugs that there cosmetic or trivial, or user errors. In contrast, in exploratory testing, you're working more like a surgeon looking for a tumor. You've seen the signs of software tumors before (for example, degradation in performance caused by a memory leak) and you put that experience to use as you probe for high-value bugs. You start your testing with an idea of what tests you want to perform, and then based on situations caused by the software's behavior, coupled with your past experience, you start to explore complex situations. This situational testing is one of exploratory testing's strengths.

So, don't feel guilty. Observe the software under test, evaluate its behavior through the filter of your experience and keep on exploring!

Ref:

[1] Exploratory Testing Explained by James Bach

[2] http://blogs.msdn.com/imtesty/archive/2007/10/19/exploratory-testing-versus-ad-hoc-testing.aspx

[3] http://www.sqaforums.com/showflat.php?Cat=0&Board=UBB46&Number=406151&page=2&fpart=all