Software Testing...and Ketchup

When you're assembling the elements of a software test "toolbox" you'll probably start with tools such as:

• Test organization/execution frameworks (such as JUnit)
• UI automation frameworks - (such as Selenium for web-based products)
• Web Service testing - (such as soapUI)
• Memory use profiling - (such as jProfiler)
• Defect tracking (such as Bugzilla)

But, you might also need to add a little ketchup.

I had just completed translating a set of human tasks to automated UI test scripts. After having run the tests in pieces and then assembled the pieces into a final revision, I thought I was finished. The test however, continually failed. 

After debugging the problem, I discovered that the failure was not due to a test coding error. As part of its initialization, the test had to run a "zip-up" a directory into a .zip file, but the test was unable to locate the "zip" binary file in the /usr/bin directory. This made no sense, as both the directory and the zip binary were clearly present on the test system. The problem was also not due to file or directory permissions as the script was able to successfully access other utilities, that were located in the same directory and were configured with the same permissions.

It took me a while to determine that the cause of the script failing was not its ability to access or execute a utility in /usr/bin, the problem was simply the number of files in that directory. 

The script, remember, was a UI test script. In order for the script to invoke the zip utility, it located the utility's binary file with a UI-based file system browser. The large number of utilities in the directory (over 2000) caused delays in the system's ability to populate the file system browser with the directory contents. The test script had the misfortune of depending on a utility, the name of which began with the letter "z" and the logic of the script was such that it assumed that the zip utility would always be displayed and could always be invoked.

My initial fix for the problem was to add a sleep statement to the script to give the file system browser the time it needed to display the entire directory listing. I later modified this to be more flexible and avoid unnecessary delays by querying the file system browser multiple times while it waited for the full directory contents to be displayed.

Once the test script was complete and running cleanly I complained about the timing issues to a co-worker. He was more amused than sympathetic and replied to me: "Well, what did you expect? You forgot the ketchup! Automation is like a hamburger, it needs some ketchup. That's what sleep statements are to automation. They are the ketchup in the recipe!"

On a practical note, the "ketchup" delay was necessary as, unlike a human being, who would pause by reflex to wait for visual feedback, my test script was originally designed to "plow ahead" at top speed, regardless of whether the UI elements on which it depended were available.

And on that delicious note, it's time to wrap up 2011! Happy 2012 everyone!

As much as I'd like to take credit for thinking up the ketchup concept, I can't. I owe it all to my friend and colleague Martin! Diky moc (many thanks) Martin! 

JBoss Survey (and a discount) at Packt

Packt, the nice folks publishing the JBoss ESB Begineers' Guide later on this month, have a new survey for JBoss users and books.

The survey is here: http://snipurl.com/21212r0

Packt is overseeing a community survey for JBoss users to find out what support they need to aid and enhance their JBoss experience. Packt want to make sure they are making the right offers to all JBoss users and are providing what they need and want.

The survey is about all of JBoss projects (Drools, AS, jBPM and more). The questions will help to provide a snapshot of how the community finds support, how they self-educate, help each other and what information they need now.

As Packt think the results will be of genuine interest to the JBoss community, the results of the survey will be published.

Not only that but every JBoss user that completes the survey will receive a 25% discount off their next JBoss eBook purchase from Packt.

The JBoss ESB Book is now in Packt's RAW Program

The soon-to-be-published (by Packt) JBoss ESB Beginner's Guide is now in the Packt "RAW" program - so that the book can be purchased pre-release - and chapters are provided as they are completed.

http://www.packtpub.com/jboss-esb-beginners-guide/book

The ESB Beginners Guide just Showed up on Amazon (Yes, THE Amazon!)

http://www.amazon.com/JBoss-ESB-Beginners-Guide-DiMaggio/dp/1849516588

It won't be long now...

Quarter Century Mark at Fotopedia!

Well, summer is about over. That's sad, but Fotopedia was nice enough to publish some of my late summer pictures:

Crawford Notch on Fotopedia

Provincetown Harbor on Fotopedia

In fact, I just hit a total of 25 articles as a top contributor!

(OK, back to software.....)

The JBoss ESB Book

The "little" project that has prevented me from having much time for blogging this summer is nearing completion.

The book will be out in November!

http://www.packtpub.com/jboss-esb-beginners-guide/book

New Pictures Voted Tops in Fotopedia

Just got some pictures of St Peter's church (no, the one in Bermuda, not Rome) voted up on Fotopedia. Very cool to see this!

St. Peter's Church, St. George's on Fotopedia

OK - the next post will be about software and testing...   

The Best Cross Posting of the Year!

One of the great things about blogging, is when your blog posts are thought of so highly by readers that they cross post them.  I was happily surprised this week to find that my post "Middleware in English Please" had been reposted. In Brazil and in Portuguese!

http://www.webbertek.com.br/site/interno.php?id_str=2&id_sub=17&id_ctns=44

Obrigado!

In praise of Micro-Reviews

The test plan was a work of art.

One hundred and twenty eight and three quarter pages of sheer brilliance.

It started with the IEEE standard and then added two bedrooms and an extra bathroom. Functional tests, stress tests, usability tests. Server backend and client frontend. Risks and mitigation strategies. Network diagrams and functional decompositions. Cross references to the functional specs, design specs, and even the team member's dietary requirements. Weeks of investigation and writing.

We proof-read it, revised it, formatted it, reformatted it, and sent it out to the entire project team for their review and feedback. And, because we knew that everyone already had a heavy workload, we allowed a full 48 hours for their review.

With as much pride in our workmanship as a ship builder, we sent it out for review and waited for the glowing reviews.

After 48 hours we received......nothing.

We decided that given the size and complexity of the plan that we would give people more time. So, we sent out an email to the entire project team and asked everyone to review the plan in 24 hours.

24 hours later we received......nothing.  

At this point, since the wholesale approach wasn't getting us anywhere, I decided to talk to one of the reviewers on a retail level. I tracked down Zyg, a senior member of the team and asked him why he hadn't reviewed the plan. His response was:

"Dude. I don't have time to read Charlie Sheen's tweets and you sent me a novel. It's like I ordered a glass of beer and you ran me over with a truck of Budweiser."

At that point, we still felt like shipbuilders, but of the Titanic.

Then, he offered an interesting deal. He said:

"Look, I'll try to help. I don't touch the GUIs, but I'll review the section that covers my server side code, so we can both get something out of it. I'll tell you where your tests suck, and maybe we can pre-empt you QE guys logging a hundred bugs where my code sucks. OK - how long is the section in that monster you wrote on my part of the code?"

So there it was. The answer was micro-reviewing.

Our goal was to collect technical information from people who were the subject matter experts for a variety of technical areas. So, why not let them focus their energies on those areas in which they were the experts? And, not burden them with having to read and review the entire work?

To be sure, this was not a perfect solution, as we still wanted input from people on the entire plan, that is, on the overall strategy and how the tests all fit together. But for that higher level view we could concentrate on the team leads and managers. And we could still divide and conquer, as we would now have review coverage for most of the discrete functional areas, thanks to our micro-review approach.

OK. That problem's solved. Now I own Zyg lunch...at the local microbrewery. 

(This is a somewhat fictionalized story - Zyg really does exist, but not in software development, dude.)

Software Metrics Bloom in the Spring

It's finally(!) spring in New England. Although, it is snowing today...not an April Fool's joke!



The flowers are opening and the ice rinks will soon be closing. Before the season completely changes, I thought it would be a good idea to revisit my friend the software manager/hockey dad.

When I asked him how things were going with his previously troubled QA team he said:

"Oh man, they're doing better and actually finding bugs, but now I have a new problem. They never stop finding bugs. Here's the deal - we planned out an 8 week project release schedule. So, we're in week 8, working on the final build before we want to ship, and they found 18 new bugs and almost all of them were in the same feature!"

At this point, I asked him, "So, what was the find/fix rate looking like?"

His response was a blank stare.  

I tried again, "Were you tracking the history of the bugs that were found in each component?"

Another blank stare.

So I told him, "Dude, you gotta get some bug tracking metrics. You need to start planning your future based on what happened in your past. You're treating your bug tracking system like a write-only database. Remember what Seinfeld's friend Newman said about the mail.  When you control it, you control information. You shouldn't just record the information, you need to start using it."

(Ok, I am fictionalizing things a bit. But, he does really talk like that, dude.)

Mining Bug Data - Gently Introducing Metrics

There's a joke in golf that goes something like this:

Q: What's the most important shot in golf?

A: The next one.

It can be the same way when you test software. Technology always changes quickly, but designs and code implementations can change more quickly. And the current state of the software under test can change even more quickly as tests expose code that is more at risk than your original plans had taken into account. It can be easy to fall into a trap of only looking forward. If, however, you make more proactive use of your bug tracking information, you can use that information to assist you both in making decisions for the present and plans for the future. What you have to do is to track some bug metrics and "mine" that information,

OK - how can you introduce metrics to a team with no experience in tracking or using them?

The short answer is "gently."

The longer answer is that you introduce a small number of metrics initially, ensure that tracking these metrics adds logos or no overhead to the team's already heavy workload, and and show the team an immediate benefit/payback from tracking the metrics.  

What are the best metrics to start with? That's easy, you can start with the metric that your bug tracking system already tracks for you.

Find Rate -vs- Fix Rate

Regardless of the software development model that you follow, you should see a pattern in the number of new bugs logged. The numbers will start slow when the QA team is more occupied with test development, rise when the tests are being debugged, peak when each test is being run for the first time, then decline sharply as test cycles continue and the only new bugs found are either in functional areas where testing had previously been blocked by other bugs or are bugs that are newly introduced when other bugs are resolved. The fix rate should follow the same general pattern, with a slight time lag.

A sign of trouble in the find rate for our hockey dad's product would have been that consistent find rate from week to week. The calendar might have told him that he was near the end of his planned schedule, but the find rate, as it was not decreasing, would have told him that the product was not complete.

What might have been the causes for this consistent find rate? Maybe the test development was not complete, so that each week new tests were being run for the first time. Or, maybe each new weekly build introduced new bugs. In order to have a clear picture of the state of the project, he would have to start tracking one more metric: the root cause of each bug.

Where is the Next Bug Coming From? 

I suggested that if it was the case that the QA team was consistently finding new bugs with old tests, that it was time for him to look beyond just the number of bugs that were being found, but to also look at the locations in the code where the bugs were being found.  

First, as to the location of the bugs, here's another software riddle;

Q:  Where are you likely to find the next 100 bugs in your product?

A: In the same places you found the last 100 bugs.

(As much as I'd like to take credit for this concept, I can't. It's from Glenford Myers in his groundbreaking book, The Art of Software Testing.)

It may be that the code in these locations is complex, which means that it is difficult to maintain so that every change runs the risk of introducing a new bug. Or, the code may have been changed and patched so many times that the original design integrity has been compromised. Or, it may be that the code is just plain buggy so that running any new tests finds new bugs.

Whatever the specifics, changes to code introduce risk as along with the code changes that are added, bugs can also be added. So, the places where you found your last bugs is often the places where you will find your next bugs.

OK - now that you have this information about where in the code you may find future bugs, what do you do with it? You can allocate more time and resources to building additional tests to give those functional areas more test coverage. And you can also ensure that any change made to the code in those areas, even a seemingly minor change is done carefully and with an eye to the likelihood of a minor bug fix inadvertently opening up a major new bug. In short, this information can give you a road map as to where in the code you should tread lightly.  

What's Next?

I didn't want to overload him with helpful suggestions all at once, but as he left, I reminded him that there is always uncertainty in software testing. But, if was able to leverage information from the past on his project, he might be able to better predict, or at least plan for the possible events in its future. And, some of the initial information that he needed to do this was already being recorded by his bug tracking system.

http://swqetesting.blogspot.com/view

Looks like the folks at Blogger have added some cool new display options.

Just try this variation on the blog URL: http://swqetesting.blogspot.com/view

And select an option under "sidebar." Very nice!

In the top 100 Software Test Blogs!

Nice surprise on an early spring morning:

http://www.vietnamesetestingboard.org/zbxe/?document_srl=487854

I'm #81!

Fotopedia - that was fast!

Just got a picture approved - for Boston's North End:

North End, Boston on Fotopedia

And Stonington, Maine:

Stonington, Maine on Fotopedia

And Truro, Cape Cod:

Truro, Massachusetts on Fotopedia

Fotopedia!

Started making some contributions to Fotopedia last night:   http://www.fotopedia.com/wiki/North_End,_Boston/candidates

Hmm - each picture needs (5) votes to be accepted. Time to start up the campaign machine...   ;-)

And - DZone picked it up too!

The JBossESB-Smooks post is here: 

http://java.dzone.com/articles/jbossesb-smooks-integration

New post in JBoss SOA Blog!

http://www.smooks.org/

Writing about Smooks  this time.

What's "Smooks?" Read on MacDuff...

http://jboss-soa-p.blogspot.com/2011/01/jbossesb-smooks-integration-in-soa.html