Software Testing...and Ketchup

When you're assembling the elements of a software test "toolbox" you'll probably start with tools such as:

• Test organization/execution frameworks (such as JUnit)
• UI automation frameworks - (such as Selenium for web-based products)
• Web Service testing - (such as soapUI)
• Memory use profiling - (such as jProfiler)
• Defect tracking (such as Bugzilla)

But, you might also need to add a little ketchup.

I had just completed translating a set of human tasks to automated UI test scripts. After having run the tests in pieces and then assembled the pieces into a final revision, I thought I was finished. The test however, continually failed. 

After debugging the problem, I discovered that the failure was not due to a test coding error. As part of its initialization, the test had to run a "zip-up" a directory into a .zip file, but the test was unable to locate the "zip" binary file in the /usr/bin directory. This made no sense, as both the directory and the zip binary were clearly present on the test system. The problem was also not due to file or directory permissions as the script was able to successfully access other utilities, that were located in the same directory and were configured with the same permissions.

It took me a while to determine that the cause of the script failing was not its ability to access or execute a utility in /usr/bin, the problem was simply the number of files in that directory. 

The script, remember, was a UI test script. In order for the script to invoke the zip utility, it located the utility's binary file with a UI-based file system browser. The large number of utilities in the directory (over 2000) caused delays in the system's ability to populate the file system browser with the directory contents. The test script had the misfortune of depending on a utility, the name of which began with the letter "z" and the logic of the script was such that it assumed that the zip utility would always be displayed and could always be invoked.

My initial fix for the problem was to add a sleep statement to the script to give the file system browser the time it needed to display the entire directory listing. I later modified this to be more flexible and avoid unnecessary delays by querying the file system browser multiple times while it waited for the full directory contents to be displayed.

Once the test script was complete and running cleanly I complained about the timing issues to a co-worker. He was more amused than sympathetic and replied to me: "Well, what did you expect? You forgot the ketchup! Automation is like a hamburger, it needs some ketchup. That's what sleep statements are to automation. They are the ketchup in the recipe!"

On a practical note, the "ketchup" delay was necessary as, unlike a human being, who would pause by reflex to wait for visual feedback, my test script was originally designed to "plow ahead" at top speed, regardless of whether the UI elements on which it depended were available.

And on that delicious note, it's time to wrap up 2011! Happy 2012 everyone!

As much as I'd like to take credit for thinking up the ketchup concept, I can't. I owe it all to my friend and colleague Martin! Diky moc (many thanks) Martin!