The Value of a Fresh Set of Eyes

The test failed.

I checked and rechecked the results. There were no errors or exceptions. The logs were all clean.

But, the test failed.

I rebooted the server and reran the test.

And the test still failed.

It's not an uncommon situation in software QE to encounter unexpected test failures, but this one was very puzzling. Everything seemed to be functioning correctly, bit the test always failed.

The test in question verified that data could traverse a complex series of actions in the software under test. In the course of being routed through the system, the data would be processed into multiple intermediate forms such as files and messages and then be sent between multiple subsystems over protocols including FTP and JMS.

"Divide and conquer" is an effective debugging technique, so I broke the test into pieces and then verified that each piece was functioning correctly. I checked to ensure that messages were correctly formatted and written to queues. I also checked to ensure that the intermediate work files, each created with a different file type, were created in the correct directory path on the various FTP servers.

And the test still failed.

Finally, after staring at this test for a few hours, I asked a co-worker to take a look at it. After about 5 minutes he laughed and said, "The files on the FTP are in the correct places and have the correct type extensions, but the files are empty." The problem was a matter of timing. The test was trying to read the files before any data was written to them.

In "The Cathedral and the Bazaar,"[1] Eric Raymond observed the principle that "many eyes make shallow bugs." He was referring to how in open source software, a program's code is freely available, and how this openness enables more people to identify and resolve a program's flaws.

It can be difficult to test your own code. After you spend time and invest effort in building a program you can take for granted how, and how well, it operates in certain circumstances. You can look at and simply not see its flaws.

It's a common, and very effective practice on software test teams to rotate assignments. This helps to ensure that no one develops a "stale" view of the software under test or begins to take any potential flaws in the software as an intrinsic part of the the code.

So, maybe we have another principle, "fresh eyes see things that you look right past."

References:

[1] http://www.catb.org/~esr/writings/homesteading/

Special thanks to Martin! Děkuji ti!