Len DiMaggio's (mostly) Software Testing Weblog

On Kindle too!

2012-01-25T20:02:00.001-08:00

And - the JBoss ESB book is on Kindle too:

http://www.amazon.com/JBoss-ESB-Beginners-Guide-ebook/dp/B0071LKMTU/ref=dp_kinw_strp_1?ie=UTF8&m=AG56TWVU5XWC2

The JBossESB Book is officially published!

2012-01-25T05:57:00.000-08:00

It's available at Packt now! http://www.packtpub.com/jboss-esb-beginners-guide/book

And at Amazon: http://www.amazon.com/JBoss-ESB-Beginners-Guide-DiMaggio/dp/1849516588/

A review of the book is here: http://rickwagner.blogspot.com/2011/10/book-review-for-jboss-esb-beginners.html

Embracing the Sliding Window of Doubt

2012-01-19T20:10:00.000-08:00

We create schedules for many different things but generally for the same reasons - to enable us to determine when in the future a task will be completed, to enable us to allocate time and money and resources to perform that task, and to enable other organizations to coordinate their future actions.

In other words, we create schedules in an attempt to predict the future.

But, why do we get it wrong so often?

Why is it that, in spite of how precisely we plan our tasks and try to take into account what might go wrong, so many project schedules miss their original target completion date?

Why did Boston's famous "Big Dig" road and tunnel construction project (http://en.m.wikipedia.org/wiki/Big_Dig_(Boston,_Massachusetts)) slip from being a $3 billion project to a $15 billion project?

Part of the reason may be simply that it is a difficult task to create a schedule that can accurately predict the future. Another part of the reason may be that people, as Frederick Brooks observed in "The Mythical Man-Month," (http://en.wikipedia.org/wiki/The_Mythical_Man-Month) are eternal optimists. We always think that "this time" things will go well, and we won't have any schedule-related problems. As a result, we fail to plan for everything that may possibly "go wrong."

But, there may be another reason. Maybe we cause our own imprecision by trying to be too precise. In other words, maybe we set ourselves up for failure and make schedule "slips" a self-fulfilling prophecy.

Which leads me to a golf story. An historical golf story, that is.

When he was asked to explain his surprising lack of success when he tried to play competitive golf a few years after his retirement at the early age 28, golf great Bobby Jones attributed it to his misguided attempt to achieve a level of precision and consistency greater than he had ever had during his best playing years. Who was Bobby Jones? The Tiger Woods of his era - back when gofers wore ties.

Maybe it's a bit like that with scheduling. Do we doom ourselves to always creating schedules that are missed by trying to achieve a target that is so precise that it can never actually be hit?

I've been thinking lately that a better approach for scheduling a large, long running, and complex project schedule is to not attempt to target a single "hard" and unchanging target completion date (for example, "our project will be completed on May 23rd at 09:00, Eastern Daylight Time"), but rather but rather to target a window of time in which the project will be completed, "slide" that window closer or further away and have it expand and contract based upon your level of confidence according to the current conditions of the project, all in the context of the the current project conditions.

This window can be thought of as the "sliding window of doubt."

OK, this idea of a "sliding window of doubt" raises a couple of questions:

What are the potential advantages of a sliding window approach?

One of the best aspects the sliding window approach is that it of scheduling a project schedule with a sliding window is that it converts the reassessment of the schedule from a reactive to a proactive task. Reassessing the schedule, and possibly sliding the project completion window is not an aberration or exception, that is only done as a reaction to a problem. Instead, it is a ongoing part of the management of the project. These re-assessments will become increasingly more accurate estimate as the project proceeds as the more you learn about the problems that a project faces during its development the better you can predict its future. As a result, you will have more accurate estimates (the window only shrinks when your confidence in the project's ability to meet its schedule increases) based on the re-assessments, and the project team will have constant access to up-to-date schedule information.

Another advantage of the sliding window approach is the flexibility that it provides the project team. The "consumers" of the schedule include the project team members themselves as well as the consumers (other project teams or end user customers) of the project deliverables, as these consumers must create their own schedules, for their own deliverables, all of which depend on your project and its schedule. With a rigid, single-date based schedule, even small changes can effect that date, and cause disruption (or maybe even panic) to dependent project and customer schedules.

Also, by building into the project plan the ability to slide the release window,people who might otherwise delay in providing bad news that would affect the schedule, out of fear of a "shoot the messenger" situation, may be more inclined to make that news available to the team sooner.

How is a "sliding" window any different from defining a more convention schedule, and then "slipping" the target dates?

The difference really comes down to what controls the schedule changes.

As we discussed just a moment ago, the constant reassessment of the schedule, where the level of doubt is re-evaluated based on the current project conditions, is a proactive task. Your project team is in more control of its fate with the sliding window approach. Also, remember that these re-assessments are not only performed when there are problems to be resolved, they are also performed when things are going well. Accordingly, while the window can slide out in response to unexpected problems, it can also move in if the project is progressing better than the original plan.

So, to sum up, when you are faced with the challenge of devising a schedule for a large, complex, long-running project, it's worthwhile to consider the "sliding window of doubt" as an approach to scheduling key milestones and release dates. At its core, this approach provides you with the flexibility to refine the schedule over time, and perform proactive schedule reassessments, that will result in the window sliding closer or further away, and expanding or contracting, all based the most up-to-date information available.

We started this discussion by asking - "Why does every project miss its target milestones and completion dates?"

Maybe the the real question that we should ask ourselves is: Why does every project schedule seem to miss its target completion date?

If the dates were rigidly defined, and the project team never re-assessed those dates as the project proceeded and unanticipated problems were encountered, then the dates were probably never achievable in the first place. What the project needed was a sliding window of doubt!

Postscript - the presence of doubt or the absence of confidence?

It's interesting how your occupation can color your view of the world. If you work in engineering (and especially if you work in software testing) you tend to look for the negative side of things. In other words, you look for what might go wrong. People engaged in sales or marketing tend to have a rosier view of things. For example, when I described the sliding window of doubt to a marketing person, she replied:

"Oh, you mean the sliding window of confidence!"

I guess she was correct, as confidence is the inverse of doubt!

(Special thanks to John Graham (http://osmusings.blogspot.com/) for his inspiration for this post.)

Software Testing...and Ketchup

2011-12-25T19:51:00.000-08:00

When you're assembling the elements of a software test "toolbox" you'll probably start with tools such as:

Test organization/execution frameworks (such as JUnit)
UI automation frameworks - (such as Selenium for web-based products)
Web Service testing - (such as soapUI)
Memory use profiling - (such as jProfiler)
Defect tracking (such as Bugzilla)

But, you might also need to add a little ketchup.

I had just completed translating a set of human tasks to automated UI test scripts. After having run the tests in pieces and then assembled the pieces into a final revision, I thought I was finished. The test however, continually failed.

After debugging the problem, I discovered that the failure was not due to a test coding error. As part of its initialization, the test had to run a "zip-up" a directory into a .zip file, but the test was unable to locate the "zip" binary file in the /usr/bin directory. This made no sense, as both the directory and the zip binary were clearly present on the test system. The problem was also not due to file or directory permissions as the script was able to successfully access other utilities, that were located in the same directory and were configured with the same permissions.

It took me a while to determine that the cause of the script failing was not its ability to access or execute a utility in /usr/bin, the problem was simply the number of files in that directory.

The script, remember, was a UI test script. In order for the script to invoke the zip utility, it located the utility's binary file with a UI-based file system browser. The large number of utilities in the directory (over 2000) caused delays in the system's ability to populate the file system browser with the directory contents. The test script had the misfortune of depending on a utility, the name of which began with the letter "z" and the logic of the script was such that it assumed that the zip utility would always be displayed and could always be invoked.

My initial fix for the problem was to add a sleep statement to the script to give the file system browser the time it needed to display the entire directory listing. I later modified this to be more flexible and avoid unnecessary delays by querying the file system browser multiple times while it waited for the full directory contents to be displayed.

Once the test script was complete and running cleanly I complained about the timing issues to a co-worker. He was more amused than sympathetic and replied to me: "Well, what did you expect? You forgot the ketchup! Automation is like a hamburger, it needs some ketchup. That's what sleep statements are to automation. They are the ketchup in the recipe!"

On a practical note, the "ketchup" delay was necessary as, unlike a human being, who would pause by reflex to wait for visual feedback, my test script was originally designed to "plow ahead" at top speed, regardless of whether the UI elements on which it depended were available.

And on that delicious note, it's time to wrap up 2011! Happy 2012 everyone!

As much as I'd like to take credit for thinking up the ketchup concept, I can't. I owe it all to my friend and colleague Martin! Diky moc (many thanks) Martin!

JBoss Survey (and a discount) at Packt

2011-12-07T11:36:00.001-08:00

Packt, the nice folks publishing the JBoss ESB Begineers' Guide later on this month, have a new survey for JBoss users and books.

The survey is here: http://snipurl.com/21212r0

Packt is overseeing a community survey for JBoss users to find out what support they need to aid and enhance their JBoss experience. Packt want to make sure they are making the right offers to all JBoss users and are providing what they need and want.

The survey is about all of JBoss projects (Drools, AS, jBPM and more). The questions will help to provide a snapshot of how the community finds support, how they self-educate, help each other and what information they need now.

As Packt think the results will be of genuine interest to the JBoss community, the results of the survey will be published.

Not only that but every JBoss user that completes the survey will receive a 25% discount off their next JBoss eBook purchase from Packt.

The JBoss ESB Book is now in Packt's RAW Program

2011-09-29T09:47:00.000-07:00

The soon-to-be-published (by Packt) JBoss ESB Beginner's Guide is now in the Packt "RAW" program - so that the book can be purchased pre-release - and chapters are provided as they are completed.

http://www.packtpub.com/jboss-esb-beginners-guide/book

The ESB Beginners Guide just Showed up on Amazon (Yes, THE Amazon!)

2011-09-12T15:24:00.001-07:00

http://www.amazon.com/JBoss-ESB-Beginners-Guide-DiMaggio/dp/1849516588

It won't be long now...

Quarter Century Mark at Fotopedia!

2011-09-02T19:28:00.000-07:00

Well, summer is about over. That's sad, but Fotopedia was nice enough to publish some of my late summer pictures:

Crawford Notch on Fotopedia

Provincetown Harbor on Fotopedia

In fact, I just hit a total of 25 articles as a top contributor!

(OK, back to software.....)

The JBoss ESB Book

2011-08-17T17:48:00.000-07:00

The "little" project that has prevented me from having much time for blogging this summer is nearing completion.

The book will be out in November!

http://www.packtpub.com/jboss-esb-beginners-guide/book

New Pictures Voted Tops in Fotopedia

2011-06-26T19:12:00.000-07:00

Just got some pictures of St Peter's church (no, the one in Bermuda, not Rome) voted up on Fotopedia. Very cool to see this!

St. Peter's Church, St. George's on Fotopedia

OK - the next post will be about software and testing...

The Best Cross Posting of the Year!

2011-06-22T17:05:00.000-07:00

One of the great things about blogging, is when your blog posts are thought of so highly by readers that they cross post them. I was happily surprised this week to find that my post "Middleware in English Please" had been reposted. In Brazil and in Portuguese!

http://www.webbertek.com.br/site/interno.php?id_str=2&id_sub=17&id_ctns=44

Obrigado!

In praise of Micro-Reviews

2011-05-18T18:47:00.000-07:00

The test plan was a work of art.

One hundred and twenty eight and three quarter pages of sheer brilliance.

It started with the IEEE standard and then added two bedrooms and an extra bathroom. Functional tests, stress tests, usability tests. Server backend and client frontend. Risks and mitigation strategies. Network diagrams and functional decompositions. Cross references to the functional specs, design specs, and even the team member's dietary requirements. Weeks of investigation and writing.

We proof-read it, revised it, formatted it, reformatted it, and sent it out to the entire project team for their review and feedback. And, because we knew that everyone already had a heavy workload, we allowed a full 48 hours for their review.

With as much pride in our workmanship as a ship builder, we sent it out for review and waited for the glowing reviews.

After 48 hours we received......nothing.

We decided that given the size and complexity of the plan that we would give people more time. So, we sent out an email to the entire project team and asked everyone to review the plan in 24 hours.

24 hours later we received......nothing.

At this point, since the wholesale approach wasn't getting us anywhere, I decided to talk to one of the reviewers on a retail level. I tracked down Zyg, a senior member of the team and asked him why he hadn't reviewed the plan. His response was:

"Dude. I don't have time to read Charlie Sheen's tweets and you sent me a novel. It's like I ordered a glass of beer and you ran me over with a truck of Budweiser."

At that point, we still felt like shipbuilders, but of the Titanic.

Then, he offered an interesting deal. He said:

"Look, I'll try to help. I don't touch the GUIs, but I'll review the section that covers my server side code, so we can both get something out of it. I'll tell you where your tests suck, and maybe we can pre-empt you QE guys logging a hundred bugs where my code sucks. OK - how long is the section in that monster you wrote on my part of the code?"

So there it was. The answer was micro-reviewing.

Our goal was to collect technical information from people who were the subject matter experts for a variety of technical areas. So, why not let them focus their energies on those areas in which they were the experts? And, not burden them with having to read and review the entire work?

To be sure, this was not a perfect solution, as we still wanted input from people on the entire plan, that is, on the overall strategy and how the tests all fit together. But for that higher level view we could concentrate on the team leads and managers. And we could still divide and conquer, as we would now have review coverage for most of the discrete functional areas, thanks to our micro-review approach.

OK. That problem's solved. Now I own Zyg lunch...at the local microbrewery.

(This is a somewhat fictionalized story - Zyg really does exist, but not in software development, dude.)

Software Metrics Bloom in the Spring

2011-04-01T19:10:00.000-07:00

It's finally(!) spring in New England. Although, it is snowing today...not an April Fool's joke!

The flowers are opening and the ice rinks will soon be closing. Before the season completely changes, I thought it would be a good idea to revisit my friend the software manager/hockey dad.

When I asked him how things were going with his previously troubled QA team he said:

"Oh man, they're doing better and actually finding bugs, but now I have a new problem. They never stop finding bugs. Here's the deal - we planned out an 8 week project release schedule. So, we're in week 8, working on the final build before we want to ship, and they found 18 new bugs and almost all of them were in the same feature!"

At this point, I asked him, "So, what was the find/fix rate looking like?"

His response was a blank stare.

I tried again, "Were you tracking the history of the bugs that were found in each component?"

Another blank stare.

So I told him, "Dude, you gotta get some bug tracking metrics. You need to start planning your future based on what happened in your past. You're treating your bug tracking system like a write-only database. Remember what Seinfeld's friend Newman said about the mail. When you control it, you control information. You shouldn't just record the information, you need to start using it."

(Ok, I am fictionalizing things a bit. But, he does really talk like that, dude.)

Mining Bug Data - Gently Introducing Metrics

There's a joke in golf that goes something like this:

Q: What's the most important shot in golf?

A: The next one.

It can be the same way when you test software. Technology always changes quickly, but designs and code implementations can change more quickly. And the current state of the software under test can change even more quickly as tests expose code that is more at risk than your original plans had taken into account. It can be easy to fall into a trap of only looking forward. If, however, you make more proactive use of your bug tracking information, you can use that information to assist you both in making decisions for the present and plans for the future. What you have to do is to track some bug metrics and "mine" that information,

OK - how can you introduce metrics to a team with no experience in tracking or using them?

The short answer is "gently."

The longer answer is that you introduce a small number of metrics initially, ensure that tracking these metrics adds logos or no overhead to the team's already heavy workload, and and show the team an immediate benefit/payback from tracking the metrics.

What are the best metrics to start with? That's easy, you can start with the metric that your bug tracking system already tracks for you.

Find Rate -vs- Fix Rate

Regardless of the software development model that you follow, you should see a pattern in the number of new bugs logged. The numbers will start slow when the QA team is more occupied with test development, rise when the tests are being debugged, peak when each test is being run for the first time, then decline sharply as test cycles continue and the only new bugs found are either in functional areas where testing had previously been blocked by other bugs or are bugs that are newly introduced when other bugs are resolved. The fix rate should follow the same general pattern, with a slight time lag.

A sign of trouble in the find rate for our hockey dad's product would have been that consistent find rate from week to week. The calendar might have told him that he was near the end of his planned schedule, but the find rate, as it was not decreasing, would have told him that the product was not complete.

What might have been the causes for this consistent find rate? Maybe the test development was not complete, so that each week new tests were being run for the first time. Or, maybe each new weekly build introduced new bugs. In order to have a clear picture of the state of the project, he would have to start tracking one more metric: the root cause of each bug.

Where is the Next Bug Coming From?

I suggested that if it was the case that the QA team was consistently finding new bugs with old tests, that it was time for him to look beyond just the number of bugs that were being found, but to also look at the locations in the code where the bugs were being found.

First, as to the location of the bugs, here's another software riddle;

Q: Where are you likely to find the next 100 bugs in your product?

A: In the same places you found the last 100 bugs.

(As much as I'd like to take credit for this concept, I can't. It's from Glenford Myers in his groundbreaking book, The Art of Software Testing.)

It may be that the code in these locations is complex, which means that it is difficult to maintain so that every change runs the risk of introducing a new bug. Or, the code may have been changed and patched so many times that the original design integrity has been compromised. Or, it may be that the code is just plain buggy so that running any new tests finds new bugs.

Whatever the specifics, changes to code introduce risk as along with the code changes that are added, bugs can also be added. So, the places where you found your last bugs is often the places where you will find your next bugs.

OK - now that you have this information about where in the code you may find future bugs, what do you do with it? You can allocate more time and resources to building additional tests to give those functional areas more test coverage. And you can also ensure that any change made to the code in those areas, even a seemingly minor change is done carefully and with an eye to the likelihood of a minor bug fix inadvertently opening up a major new bug. In short, this information can give you a road map as to where in the code you should tread lightly.

What's Next?

I didn't want to overload him with helpful suggestions all at once, but as he left, I reminded him that there is always uncertainty in software testing. But, if was able to leverage information from the past on his project, he might be able to better predict, or at least plan for the possible events in its future. And, some of the initial information that he needed to do this was already being recorded by his bug tracking system.

http://swqetesting.blogspot.com/view

2011-03-31T18:18:00.000-07:00

Looks like the folks at Blogger have added some cool new display options.

Just try this variation on the blog URL: http://swqetesting.blogspot.com/view

And select an option under "sidebar." Very nice!

In the top 100 Software Test Blogs!

2011-03-17T20:55:00.000-07:00

Nice surprise on an early spring morning:

http://www.vietnamesetestingboard.org/zbxe/?document_srl=487854

I'm #81!

Fotopedia - that was fast!

2011-02-08T18:52:00.000-08:00

Just got a picture approved - for Boston's North End:

North End, Boston on Fotopedia

And Stonington, Maine:

Stonington, Maine on Fotopedia

And Truro, Cape Cod:

Truro, Massachusetts on Fotopedia

Fotopedia!

2011-02-06T13:33:00.000-08:00

Started making some contributions to Fotopedia last night: http://www.fotopedia.com/wiki/North_End,_Boston/candidates

Hmm - each picture needs (5) votes to be accepted. Time to start up the campaign machine... ;-)

And - DZone picked it up too!

2011-01-17T08:58:00.001-08:00

The JBossESB-Smooks post is here:

http://java.dzone.com/articles/jbossesb-smooks-integration

New post in JBoss SOA Blog!

2011-01-12T15:06:00.000-08:00

http://www.smooks.org/

Writing about Smooks this time.

What's "Smooks?" Read on MacDuff...

http://jboss-soa-p.blogspot.com/2011/01/jbossesb-smooks-integration-in-soa.html

It’s Always Cold in a Room Full of Ice

2010-11-29T17:34:00.000-08:00

This story is true. Well mostly...)

One of the social benefits of being a hockey parent is being able to commiserate with other hockey parents about the shared experiences of early mornings, long drives, and always too cold ice rinks. At a recent game - at 7:00 AM on a Saturday when the temperature inside the rink was a balmy 34F - I happened to ask a fellow hockey father about his software company. His situation, and my suggested actions for him form the basis of this post. I'm hoping that this post will help him and others.

(I'm guessing that anyone involved in software testing will have encountered one or more of the types of problems that he described - but hopefully not all at the same time! Note that all the names have been changed from the humans involved to the names of household pets.)

The Horror

When I asked Walter how his company was doing, his response sounded like a cross between a hi-tech project plan, James Joyce, and Conrad's "The Heart of Darkness."

He started by saying, "The horror, man, the horror of this situation is freaking insane. "

He then described how, through an acquisition by his company, he had inherited a QE team. Walter had several years of project management experience, and while he had worked with QE teams and QE managers, he had never before had direct control over a QE team. To call the QE team "troubled" was something of an understatement. In only his first few days being responsible for the team he quickly learned the following about the team:

The team was geographically dispersed. Walter was based in Boston and the rest of the team was spread out between Baltimore, Boise, Belarus, and Bangalore. (The team had been nick-named the "killer-B's.)
Most of the team member had never met each other. The QE manager (her name was Celia) was based in Boise and had "issues" with either getting up early or staying up late to accommodate other team members' time zones. As a result, the team seldom had group meetings and the team manager rarely communicated over the telephone directly with individual team members in one-on-one conversations.
There was no mapping of tests and test coverage to product features.
A central/shared bug tracking system was used, but no organized bug triage reviews were ever held. Hundreds of old unresolved bugs or bugs that were likely no longer relevant were cluttering the bug database.
Many tests were automated, but there was no central/shared repository for all the tests. Individual team member often kept the automated tests in their personal home directories.
As the automated tests were written in multiple different languages, there was also no central/shared automated test framework. Some team member used Junit or similar frameworks, while others built their own "home grown" test frameworks.
Many of the automated tests were suspect due to a lack of maintenance. Individual engineers maintained personal lists of tests that were "known to fail."
The level of confidence in the QE team on the part of the other project teams was low. One product manager had recently asked Walter if testing was “strictly necessary, after all we do code reviews.”
Test plans that defined a test strategy or that provided traceability of product features and operational requirements to actual test coverage were non-existent. At the start of a project release, the team manager would send emails to team members instructing them as to which tests to develop. In the absence of these emails, individual team members often created whatever tests they felt were necessary.
There was no central/shared repository of archived test results. In short, he knew that he and the team were in bad shape, but because of the general lack of "institutional memory" (e.g., test plans, test results) he didn't even know how bad off he was in any detail.

Yes, It Can Always Get Worse

Years ago, before they broke the "Curse of the Bambino" (http://en.wikipedia.org/wiki/Curse_of_the_Bambino), when I commented to a friend that the prospects for the Boston Red Sox could not get any worse, an older friend of mine commented that "things can always get worse." This can be true of software too.

A few days after Walter first described his situation to me, he mentioned to me that, "Would you believe it? My QE manager just walked out!"

At this point, I thought that it was worthwhile to try to give him some advice. Besides, I had been looking for a good topic for my software testing blog for weeks. ;-)

The Wine is Bad, Throw it Out!

What I suggested to him was that he should look at the hasty exit of his QE manager as a positive development. The team had gotten itself into such a deep hole, that only some drastic changes could improve the situation.

There’s a really great line from a really bad old movie, “The Agony and the Ecstasy.” In the film, Michaelanglo (played by Charlton Heston. And if that seems like strange casting, how about Rex Harrison as the Pope?) is disgusted by his first attempt to paint the ceiling of the Sistine Chapel. When he witnesses his local bartender throwing away wine that has gone bad, he is inspired to throw away his work and start again.

Walter’s initial reaction to his team’s problems was to try to patch things together. When I asked him if the team had ever been effective or even functional, he replied, “No way man. All I keep hearing is that the team has always been a disaster.” I suggested that instead of patching things together to back to where they were before the manager left, he would better off trying to address and resolve some of the team's problems. What ever they had been doing was simply not working. He was pressed for time, as his product release schedule was tight. But, I suggested that investing only a few days in restructuring things could pay some immediate results.

While he thought he was dealing with an infinite number of problems, I suggested that he classify the problems, and the solutions, into categories:

People
Tools and Assets
Processes

The People

My diagnosis for his team’s primary problem was: fragmentation. In fact, the team was suffering from two distinct forms of fragmentation.

The first form was the fragmentation of their work effort.

Regional Fragmentation - The teams’ tasks were divided by geographic region. The stress testing was performed by the Boise team, while the UI testing was performed in Bangalore, and the integration testing was performed in Baltimore. As a result, it was common for team members, when they were asked about specific tests, would just say, “oh, the other guys do that work - those tests are theirs.” This made test status information collecting difficult if the team in one location had gone home for the day. I suggested to Walter that the work be divided such that each location would have some people able to run any type of test that the team built. In other words, spread the work tasks across the timezones and divide the team by tasks, not by geography.
What’s Your Role in this Organization? What’s My Role? - In listening to Walter describe the manner in which the recently departed manager led the team, I imagined that the general lack of information, planning, and leadership would have caused many team members to be uncertain of their role within the team. I suggested that Walter not define their roles in terms of an org-chart, but rather, to define everyone’s role in terms of their dependencies and their deliverables. The functioning of a large, dispersed team can be thought of as being analogous to integrating software modules together. What you need is not just an org-chart, but also a dependency diagram. The inter-dependencies between the team members are in effect a social contract that binds the team together.

The second form was the fragmentation of their work “community.” I think it was Robert Kennedy who lamented that divisions in society meant that “we share a city, but not a community.” In the case of Walter’s team, their geographic divisions made it difficult for the team to function as a team. Walter could not move everyone to the same physical location, but I did have some suggestions to get the team members working together better:

“They Aren’t Remote, You Are - It’s only a matter of words, but I suggested that Walter stop referring to the team members that were not in his physical location as “remote.” The fact was, that purely in terms of numbers, that the larger concentrations of team members were in locations other than Walters. If anyone was “remote,” it was Walter. The goal of this change in terminology would be to help team members feel that they were full partners in the team, and not second class team members.
Is it Time for the Meeting? What Time is it Exactly? - The practice of having the same team members be inconvenienced by the time selected for team meetings had to stop. Walter had to set up a system where the times for the meetings would rotate so everyone would share the burden of getting up early, or staying up late. And, speaking of time, I suggested that Walter always refer to the time of day for meetings or other events in UTC time, and not in his local timezone. By using UTC time, everyone on the team could share a common “virtual time zone.”
Daily Contact - Question: How do you communicate with people? Answer: You talk to them. Having regular weekly team meetings would help the team communicate, but I suggested that it was important for Walter to establish, and then work hard to maintain, daily contacts, whether on the telephone, or via on-line chat. Email, I suggested, would not be a substitute for some informal, and at least daily contact.
What’s a “Day?” - Finally, I suggested to Walter that he re-think what a “day” is. The reality of his geographically dispersed team was that, at almost any given time of day, some members of his team would be working. There really was no “end of the day.” I suggested to Walter that he not look at this as a handicap, but as an advantage. The analogy that I used was that of a ship at sea. While the ship has to keep moving 24 hours a day, the sailors don’t stay awake for 24 hours. They work in shifts. His team could do the same thing if, as we discussed a few minutes ago, their work was partitioned so that people in different locations worked together on the same tasks.

The Assets

When I asked Walter about the tests that the team created, he described them by saying, “Man, they have thousands of tests. But lots of them fail, others are disabled, and, nothing is described in any test plan documents. I have no idea if we have enough tests, too many, or no way near enough tests. And, what’s worse, no one can explain to me just what the tests actually do.”

Walter then described how he wanted to review every test, and build up documentation on the operation of each test, so that he could get at least a general idea of the feature coverage provided by the tests. He also mentioned that the project did not have any detailed design specifications, or requirement definition documents. I suggested a slightly different approach:

Define the Product’s Test Needs First - I suggested to him, that before he started wading through all the tests, he first needed to create a functional definition of the product under test. Before he started to measure test coverage, he needed to create the “yardstick” against which he could measure the coverage. This “functional decomposition” of the product would define the functions performed by the product and the user requirements that the product was intended to fulfill. AND, I suggested strongly to him that in order to make this product functional decomposition useful, it would have to include the relative priorities of each product function, and the perceived risk of bugs being found in testing each feature. My reasoning was that since it is never possible to test the infinite number of possible function, configuration, sequence of user actions, etc., it’s important to concentrate your always finite test resources on those product functions that have the highest priority and are most at risk.

In Walter’s case, since he was starting with a blank slate for a product test coverage definition, the place he had to start was with these high priority functions, configurations, integrations, use cases, etc. Once he had this definition in hand, he could then begin the review of the tests, and leverage whatever partial information individual team members had and map that information into a test coverage matrix. And then, he could expand on that definition to include features of lesser priorities until his test coverage matrix was complete.

This matrix would be the beginning of building his team’s “institutional memory.” In a geographically dispersed team, where it would frequently be difficult to contact people in real-time, it was vital that every team member have access to persistent information on the tests’ goals and coverage and the test strategy, including the defined priorities for tests. The documents would have to be able to stand on their own.

Plan is Also a Verb - The next step would be the creation of test plans. When I asked Walter about the teams’ test plans, he told me, “Man, they are terrified of writing formal plans. All they do is tell me that they don’t have time to create huge documents.”

My reply was, “Then, start by making the plans small. The goal of writing a test plan is not just a document. The act of writing the document forces people to think and review their analysis of the product’s risks.” I then suggested that Walter try a light-weight template for test plans. And, luckily, I had one handy - http://swqetesting.blogspot.com/2007/12/when-less-may-be-more-lighter-weight.html

I suggested to Walter that he introduce the team to the value of test plans by explaining to them that the plan’s real goal would be to get the team to be able to answer a series of questions. And that it was easier to ask themselves these questions before other people did! Each the answer to each question would take the form of a section of the plan. Some of these questions would be:

Introduction - what are we doing?
Test Strategy - how are we doing it
Test Priorities - what's most important?
Scope - What's being tested?
Scope - What's beyond the scope of testing?
Test Pass/Fail Criteria - how do we know that it's good or bad?
Test Deliverables - What are the docs and tests that we'll build?
Test Cases - What does each test do?
Responsibilities - who's doing what?
Schedule/Milestones - when are we doing it?
Risks and Contingencies - what might go wrong and how we'll handle it?
Approvals - do we agree?
References - pointers to background docs?
Revision history - why did the plan change and how?

The answers to these questions would be important to other teams on the project, to inform them of the test coverage planned, and to be used as a vehicle for them to provide input and suggestions and criticisms to the team.

The Processes

Finally, we talked about the tests themselves and how they were run. I had two suggestions for Water:

Open Things Up - One of the problems that Walter’s team had was a lack of information sharing. Sometimes by design and sometimes by accident team members kept important information from other team members. I suggested that Walter start to run the team as if it were an open source software project. In his essay “The Cathedral and the Bazaar,” (http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/), Eric Raymond defined Linus’ Law (named for Linus Torvalds) as "given enough eyeballs, all bugs are shallow." This is one of the great strengths of open source software; as the bugs are not hidden, they can be identified and resolved. For Walter’s team, the test plans, test results (logs, reports, etc.), and especially the tests themselves should be shared repository, where all the team members, and the people working on the Development, Management, Documentation teams, could access and review them. The reviews of the tests could be the beginning of the process of getting the tests to running on a single test framework. And, if more people on the team were aware of the automated tests’ design, then maybe someone other than the original author would be able to maintain them - AND - maybe start to clean up that backlog of old bugs too!
You’re Not Finished - You’re Just Starting - Finally, I suggested to Walter, that even if he were able to improve the effectiveness of his team with any of my suggestions, that his work was only just beginning. What’s the most important process for a software test team to adopt? To me, it’s continuous improvement. (http://swqetesting.blogspot.com/2009/12/choosing-kaizen-over-cheez-whiz.html) Software will always include bugs, and some bugs will be missed in testing. You have to analyze your mistakes in an honest manner, and constantly refine your processes to incorporate corrections to past mistakes AND adapt your processes to meet new situations. But, in order to be able to do this, you first have to document your plans and your results so that you can review them at a later date.

Closing Thoughts

Well Walter, I hope these ideas can help you dig your team out of that hole. In thinking about your situation, the word that kept coming to mind was “fragmentation.” Your team was fragmented, the tools and data stores they built and relied on were fragmented, and their working environment was fragmented. Whatever you can do to replace this fragmentation with a coordinated effort, where everyone understands their role within the team, should improve things quickly. How should you begin? Talk to your team, both as a group in a team meeting, and as individuals. Define their roles in terms of their dependencies and deliverables. Map the product features, and then map the tests to cover those features. Make all communications, and institutional memory shared and open. And, don’t fall behind by standing still. If think you see some improvement, then press for more.

Oh, and remember to think in UTC time. Just not for the hockey games! ;-)

From Panoramio to Google Earth

2010-10-05T19:31:00.001-07:00

Hey - google earth just accepted some of my pictures - http://www.panoramio.com/user/5031103 - including the orange dinosaur... ;-)

Cross Posting Times Two

2010-09-27T17:46:00.000-07:00

Just had a blog post cross-posted:

In the JBoss SOA Platform blog:

http://jboss-soa-p.blogspot.com/2010/09/asserting-security-jbossesb-
integration.html

And at DZone:

http://java.dzone.com/articles/asserting-security-jbossesb

Twitter!

2010-09-16T10:44:00.000-07:00

Finally joined twitter:

http://twitter.com/ldimaggi314

And there are even some followers already! ;-)

Cross-posted to DZone

2010-08-17T12:22:00.000-07:00

The post on the JBoss SOA Platform's Rules-JBossESB integration - was just reposted to DZone:

http://java.dzone.com/articles/jbossesb-drools-integration

Named a DZone "Most Valuable Blogger"

2010-07-19T10:38:00.000-07:00

I was just named to the DZone "MVB" list: http://www.dzone.com/aboutmvb

DZone is a great site - this is a very nice surprise on a Monday morning!

Cross Posted!

2010-07-08T08:21:00.000-07:00

The previous post - on the JBoss SOA Platform's Rules-JBossESB integration - has been re-posted to the JBoss SOA Platform blog:

http://jboss-soa-p.blogspot.com/2010/07/jbossesb-drools-integration-in-jboss.html

The JBossESB-Drools Integration in the JBoss SOA Platform

2010-06-22T18:56:00.000-07:00

As I've mentioned in previous posts and articles, one of the great strengths of the JBoss SOA Platform is the large number of integrations that it supports. Some of these integrations take the form of support for JBoss and third-party supplied JMS and UDDI providers or support for multiple JDKs and databases. The SOA Platform also supports an integration to jBPM for business process orchestration, and JBDS for application development.

In this post, I want to take a look at the SOA Platform's integration with JBoss Rules. In an earlier blog post (http://jboss-soa-p.blogspot.com/2009/07/when-content-knows-way-content-based.html), I discussed using Rules to implement content based routing over the JBossESB. In this post, we'll look at another aspect of the JBossESB-Rules integration; the creation of rules-based services.

Before we examine this integration in detail, let's take a quick look at JBoss Rules.

Drools and Rules

JBoss Rules as packaged in the SOA Platform is the commercialized version of the open source "Drools" project. (You can learn more about Drools at the project web site here: http://www.jboss.org/drools/) Drools is a unified and integrated solution for Business Rules, Business Processes Management, and Event Processing. The Drools project is organized into a number of sub-projects. When we refer to JBoss Rules in the SOA Platform, what we're primarily talking about is the Drools "Expert" sub-project. This sub-project consists of the Rules API, the Rules engine, and Rules editing and debugging tools.

Rules-based programming, as its name implies, is built on the ability to define decision points and keep them separate from other program logic. OK, that sounds interesting, but why would I want to use this? In other words, what's the big deal? Here are two reasons why this is important:

First, it enables you to separate your application's business logic and decision point handling from the application code. This means that your business process specialists can concentrate on the business logic rules and your programmers can concentrate on the application code. This makes application development and maintenance easier and more effective.
Second, and don't take this as a personal criticism of your programming skills or mine, but the since the rules engine is designed and optimized to process rules, it is more efficient than any massive if-then-else statement that you can write. So, your application's performance can be improved. The JBoss Rules engine makes use of the Rete algorithm http://en.wikipedia.org/wiki/Rete_algorithm for efficient Rules processing.

When you define a rule, the model that you follow is not "if-then-else," but rather "when" and "then." The two constructs that you use in a rule are:

The Condition - This is the left hand side of the rule and covers the "when" aspects of the rule.
The Consequence - This is the right hand side of the rule and covers the "then" aspects of the rule.

Rules are written in the Drools Rule Language (drl). This language can be extended into a Domain Specific Language (dsl) to support application-specific requirements such as medical or financial procedures and terminology.

The general template for a rule is:

1:  rule “a simple rule”
2:  when (LHS)
3:    you need a rules-based app
4:  then (RHS)
5:    build it with JBoss Rules

How does a rule access information? Through Rules' working memory. The information that rules perform operations take the form of Java beans that are referred to as "facts." (The elements in these facts are accessed through the getter and setter methods.) What happens is that facts are inserted into working memory, updated in working memory, or removed from working memory, those changes to the facts in working memory can cause the Rules' "when" conditions to be true and the rules to be executed.

It's important to note that unlike procedural programming, changes to facts can cause more than one rule to reach true conclusions and be available to be executed at once. What happens then? Well, instead of hardcoding a sequence of rules, the rules engine adds each matching rule to its "Agenda" of rules to be executed. If the Agenda includes more than one rule, the rule engine performs conflict resolution on the rules and determines the sequence in which the rules should be executed. This conflict resolution is based on the rules' salience (you define this as a property when you write the rules), how often the rule has fired in the past, complexity (the more complex a rule the more likely the rule engine will consider it to apply to the current situation), and the order in which the rules are loaded.

Invoking Rules from a JBossESB Service in the SOA Platform

The Rules - JBossESB integration in the SOA Platform enables you to access rules from an service's actions. This is supported by the org.jboss.soa.esb.actions.BusinessRuleProcessor and the org.jboss.soa.esb.actions.DroolsRuleService action classes.

The BusinessRuleProcessor class uses rules loaded from rules files. Generally, you use this class for simple rules services as loading large numbers of rules from large numbers of rules files is difficult to manage and not efficient.

For production environments, where you will have complicated rules services that deal with hundreds or even thousands of rules, it's better to use the DroolsRuleService. This service uses the RuleAgent to either access packages of rules from files, or from a Business Rules Management System (BRMS) .

The JBoss BRMS Platform (http://www.jboss.com/products/platforms/brms/) combines a central repository for rules, with a web based rules authoring interface, and rules management that provides import/export/archiving, audit trail or edits, error log, automated test development and execution, rules analysis, status tracking, and version control. The BRMS Platform rules authoring "guided editor" enables non-programmers to more easily create rules while the Platform makes it easy for rules administrartors can maintain large numbers of rules and control their development and use by users. Here's a screenshot:

The SOA Platform supports rules services that are either stateless or stateful. In the stateless model, messages sent to the services contain all the facts to be inserted into the rules engine working memory, before the rules are executed. In the stateful model, where the execution of the rules may take place in a session over an extended time period, several messages may be sent to a rule service, and the rules may fire and update either the message or the facts until a final message causes the service to end the session. The best way to explain and illustrate the Rules - JBossESB integration in the SOA Platform is with one of the Platform's "quickstart" example programs. Let's take a look.

Rules Services in Action - The Quickstart

I never get tired of saying that one of the great features of the SOA Platform is its extensive, and always growing, set of "quickstart" programs. These programs go far beyond being simple examples as they clearly illustrate various features supported by the Platform. They also serve as a great resource for writing your own applications. For our example, we'll look at the "business_rules_service" quickstart. Before we walk through the configuration and execution of the quickstart, let's take a look at its (3) Rules files. It important to note that Rules are actually used in multiple ways in this quickstart as it simulates a customer making a purchase from an e-commerce site. The quickstart uses Rules to:

Calculate the priority of an incoming customer order
Calculate the discount to be applied to an order
And route the order to the appropriate service, based on the content of the order

It's also important to note that the quickstart illustrates how a message can be modified as it is processed through the SOA Platform's JBossESB action pipeline. Let's start with the "MyBusinessRules.drl" file, as it establishes the priority of the order. (Oh, wait. We'll jump ahead a bit here and explain that by default, every incoming order has a priority value of "1.")

1:  package com.jboss.soa.esb.routing.cbr
2:
3:  #list any import classes here.
4:  import org.jboss.soa.esb.message.Message;
5:  import org.jboss.soa.esb.message.format.MessageType;
6:  import org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.OrderHeader;
7:  import org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.Customer;
8:
9:   global java.util.List destinations;
10:
11:
12:
13:   rule "Logging"
14:   when
15:     order: OrderHeader()
16:     customer: Customer()
17:   then
18:     System.out.println("Customer Status: " + customer.getStatus());
19:     System.out.println("Order Total: " + order.getTotalAmount());
20:   end
21:
22:   rule "Customer Platinum Status"
23:    when
24:     customer: Customer(status > 50)
25:     order: OrderHeader(totalAmount > 50)
26:    then
27:     System.out.println("Platinum Customer - High Priority");
28:     order.setOrderPriority(3);
29:   end
30:
31:   rule "Customer Gold Status"
32:    when
33:     customer: Customer(status > 10, status <= 50)
34:     order: OrderHeader(totalAmount > 25)
35:    then
36:     System.out.println("Gold Customer - Medium Priority ");
37:     order.setOrderPriority(2);
38:   end
39:

Let's examine this Rules file line-by-line:

Line 1 - Similar to Java, a package is a related set of rules.
Lines 4-7 - These imports perform the same function as Java language imports. Note that we're importing both SOA Platform Message related packages and packages contained in the quickstart itself.
Line 9 - Make a mental note of this global definition as it will be used by the other Rules files and by the quickstart for content based routing.
Line 13-14 - The name for the first Rule and the start of its "when" clause.
Line 15 - This creates a Rule variable named "order" and initializes with the value of the OrderHeader fact (org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.OrderHeader) that is presented to the Rule.
Line 16 does the same for a Rule variable named customer and the Customer fact. Note that there is nothing conditional about these assignments. The rule will match every OrderHeader and Customer. This matching is actually an important concept to keep in mind as part of each "when" clause involves trying to match the facts passed to the rules.
Line 20 - A rule must always have an end statement.
Line 22 - This rule sets the status for the highest priority ("Platinum") customers.
Line 24-25 - Note the differences in these statements to those in lines 15 and 16. The rule will only be executed if:

Line 24 - The customer's status is greater that 50. If this is not the case, then the customer variable we set in line 16 will not be changed, and
Line 25 - The total amount of the order is also greater than 50. If this is not the case, then the order variable that we set in line 15 will not be changed.

Line 28 - Remember how I said that the default priority for each order was set to "1". Here's where we set the priority. Note that we are setting the value in the "order" Rule variable.
Lines 31-38 - This rule sets the status for the 2nd highest priority ("Gold") customers. What's interesting in this rule is line 33 as it includes two criteria that must both be met in order for the fact to match the rule.

The next Rules file that we'll look at picks up where MyBusinessRules.drl leaves off. This Rules file is named: MyBusinessRulesDiscount.drl

1:  package com.jboss.soa.esb.routing.cbr
2:
3:  #list any import classes here.
4:  import org.jboss.soa.esb.message.Message;
5:  import org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.OrderHeader;
6:  import org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.Customer;
7:
8:  global java.util.List destinations;
9:
10:
11:
12:  rule "Logging"
13:  salience 10
14:  when
15:    order: OrderHeader()
16:    customer: Customer()
17:  then
18:    System.out.println("Customer Status: " + customer.getStatus());
19:    System.out.println("Order Total: " + order.getTotalAmount());
20:  end
21:
22:  rule "Customer Platinum Status"
23:  salience 20
24:   when
25:    customer: Customer(status > 50)
26:    order: OrderHeader(orderPriority == 3)
27:   then
28:    System.out.println("Platinum Customer - High Priority - Higher discount");
29:    order.setOrderDiscount(8.5);
30:  end
31:
32:  rule "Customer Gold Status"
33:  salience 20
34:   when
35:    customer: Customer(status > 10, status <= 50)
36:    order: OrderHeader(orderPriority == 2)
37:   then
38:    System.out.println("Gold Customer - Medium Priority - discount ");
39:    order.setOrderDiscount(3.4);
40:  end

Lines 1-8 and the "Logging" rule should look familiar, so let's move on to the other rules defined in this file.
Line 13 - Remember how we talked about how rules are not executed in the exact sequence in which they can be viewed in a rules .drl file? This rule is assigned a salience property value of 10. Since this rule only prints out some logging information, it's assigned a lower salience than the other rules defined in the file.
Line 22 - This rule sets the discount level for the highest class of customers.
Line 23 - And, since we want to be sure that this rule fires, we assign it a salience property value of 20.
Lines 25-26 - This "when" clause is true when the rule is able to match both a Customer fact with a status greater than 50 and an OrderHeader fact with an orderPriority equal to 3. If both these conditions are true, then rule the customer and order variables are initialized from the Customer and and OrderHeader facts and the rule is fired.
Line 29 - Note that when the rule fires, the setOrderDiscount setter method is executed on the rule variable "order." The same setter method is executed on the OrderHeader fact in working memory. (Remember how we talked about how rules can both react to changes to facts in working memory and also cause changes on those facts? This is an example.) For the "Platinum" class of customers, we assign a generous discount.
Lines 32-39 - This rule follows the same pattern as the "Customer Platinum Status" rule. Note that we give the "Gold" class of customers a somewhat less generous discount. ;-)

The third and final Rules file used by the quickstart controls the content-based routing used by the quickstart to route messages to services. In the SOA Platform, Rules is one of the supported mechanisms for implementing content based routing. In contrast to more static routing approaches, this form of message routing is relies on the content in the messages to dictate the route that a message takes. (For background on content based routing, please refer to this blog post: http://jboss-soa-p.blogspot.com/2009/07/when-content-knows-way-content-based.html )

This rules file is aptly named: MyRoutingRules.drl

1:  package com.jboss.soa.esb.routing.cbr
2:
3:  #list any import classes here.
4:  import org.jboss.soa.esb.message.Message;
5:  import org.jboss.soa.esb.message.format.MessageType;
6:  import org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.OrderHeader;
7:
8:  #declare any global variables here
9:  global java.util.List destinations;
10:
11:
12:   rule "Highest Priority Orders"
13:
14:       when
15:           OrderHeader( orderPriority == 3 )
16:       then
17:           System.out.println("HIGHEST PRIORITY");
18:           destinations.add("SuperSpecialCustomerService");
19:
20:   end
21:
22:   rule "Medium Priority Orders"
23:
24:       when
25:           OrderHeader( orderPriority == 2 )
26:       then
27:           System.out.println("Medium Priority");
28:           destinations.add("SpecialCustomerService");
29:   end
30:
31:   rule "Low Priority Orders"
32:
33:       when
34:           OrderHeader( orderPriority == 1 )
35:       then
36:           System.out.println("Low Priority");
37:           destinations.add("RegularCustomerService");
38:   end
39:

Lines 1-6 - Once again, these import lines should look familiar.
Line 9 - Make note of the destinations List. We'll see this used when the messages are routed.
The rules in this rules file are pretty straight-forward. In each of the three rules we add the appropriate destination, based on the orderPriority in the OrderHeader, where that destination is a service that the quickstart deploys to the JBossESB in the SOA Platform.

OK, those are the rules that we'll use in the quickstart. Let's now step through the quickstart as it is run.

To deploy the quickstart, execute this ant target:

1:  ant deploy

And, we then see this written to the server log:

1:  22:20:57,568 INFO [QueueService] Queue[/queue/quickstart_Business_Rules_Request_GW] started, fullSize=200000, pageSize=2000, downCacheSize=2000
2:  22:20:57,580 INFO [QueueService] Queue[/queue/quickstart_Business_Rules_Request_ESB] started, fullSize=200000, pageSize=2000, downCacheSize=2000
3:  22:20:57,621 INFO [QueueService] Queue[/queue/quickstart_Business_Rules_ConciergeManager] started, fullSize=200000, pageSize=2000, downCacheSize=2000
4:  22:20:57,632 INFO [QueueService] Queue[/queue/quickstart_Business_Rules_DistributionManager] started, fullSize=200000, pageSize=2000, downCacheSize=2000
5:  22:20:57,643 INFO [QueueService] Queue[/queue/quickstart_Business_Rules_BasicShipping] started, fullSize=200000, pageSize=2000, downCacheSize=2000
6:  22:20:57,682 INFO [EsbDeployment] Starting ESB Deployment 'Quickstart_business_rules_service.esb'

And to run it, exceute this ant target:

1:  ant runtest

When the quickstart is run, here's what happens, step by step. Note that while we'll be examining most of the contents of the jboss-esb.xml file in detail, we'll be doing it in pieces or fragments so that it's easier to follow. The line numbers in each of these fragments will therefore be different from the actual (whole) file.

Step 1 - Create a Message and Pass it Through a Gateway to the Deployed Quickstart Application

Like many of the SOA Platform quickstarts, the business_rules_service quickstart initiates its actions when a message is inserted into a queue that is being watched by a gateway listener. What's a gateway? On the JBossESB in the SPA Platform, everything is either a service that generates or consumes messages, or a message. That is, a message that is in the form (org.jboss.soa.esb.message) that the ESB understands. Services that can understand messages in this form are referred to as being "ESB-aware."

How can you connect other, and potentially older, legacy applications over the ESB? By using gateways. A gateway (org.jboss.soa.esb.listeners.gateway) is a service that acts as a bridge between an ESB-aware and an ESB-unaware client and service. Gateways translate information between ESB and non-ESB message formats and EPRs. (EPR stands for endpoint reference.) Gateways are listener processes in that they "listen" for incoming communications. They are different from ESB-aware listeners in that they accept data in different formats such as objects in files or SQL tables. ESB-aware listeners can only accept messages in the org.jboss.soa.esb.message format.

The SOA Platform supports these gateways:

file gateways: local filesystem, ftp, sftp and ftps
JMS
HTTP/HTTPS
email (POP3)
SQL table
Hibernate

In the case of this quickstart, we'll use a JMS gateway to receive the JMS message. The gateway queue, and its corresponding ESB-aware queue are defined in the jms-provider section of the jboss-esb.xml file:

1:  <jms-bus busid="quickstartGwChannel">
2:    <jms-message-filter dest-type="QUEUE"
3:      dest-name="queue/quickstart_Business_Rules_Request_GW" />
4:  </jms-bus>
5:  <jms-bus busid="quickstartEsbChannel">
6:    <jms-message-filter dest-type="QUEUE"
7:      dest-name="queue/quickstart_Business_Rules_Request_ESB" />
8:  </jms-bus>

And the listener is defined at the top of the "Business_Rules_Service" service definition:

1:   <service category="Business_RulesServices"
2:     name="Business_Rules_Service" description="The main entry point">
3:     <listeners>
4:       <!-- Gateway -->
5:         <jms-listener name="TheGateway"
6:           busidref="quickstartGwChannel" is-gateway="true" />
7:         <jms-listener name="TheESBChannel"
8:           busidref="quickstartEsbChannel" >
9:         </jms-listener>
10:     </listeners>
11:     <actions mep="OneWay">

Note that on line 6, we identify the listener as a gateway.
Also note that on line 11, we define the mep, or "message exchange pattern." In the case of this quickstart, the pattern is "OneWay" which indicates that the message pattern is asynchronous. We're sending messages, but not waiting around (or blocking) for a response.

How do we generate this message? Take a look at the "runtest" target, specifically the classname, in the quickstart's ant build.xml file:

1:  <target name="runtest" depends="compile"
2:    description="willl receive JMS message to tigger the actions in the ESB">
3:    <echo>Runs Test JMS Sender</echo>
4:    <java fork="yes" classname="org.jboss.soa.esb.samples.quickstart.businessrules.test.SendJMSMessage" failonerror="true">
5:      <classpath refid="exec-classpath" />
6:    </java>

That's right - we're sending a JMS message with a program named "SendJMSMessage." It's hard to get simpler than that. ;-)

In order to simulate a realistic customer order in the message, SendJMSMessage builds the message from the quickstart's SampleOrder.xml file:

1:  <Order orderId="1" orderDate="Wed Nov 15 13:45:28 EST 2006" statusCode="0"
2:  netAmount="59.97" totalAmount="64.92" tax="4.95">
3:      <Customer userName="user1" firstName="Harry" lastName="Fletcher" state="SD"/>
4:      <OrderLines>
5:          <OrderLine position="1" quantity="1">
6:              <Product productId="364" title="The 40-Year-Old Virgin " price="29.98"/>
7:          </OrderLine>
8:          <OrderLine position="2" quantity="1">
9:              <Product productId="299" title="Pulp Fiction" price="29.99"/>
10:          </OrderLine>
11:      </OrderLines>
12:  </Order>

The SendJMSMessage class reads this file, creates a message object of type javax.jms.ObjectMessage (remember, this is an ESB-unaware message), and writes it to the queue (queue/quickstart_Business_Rules_Request_GW) on which our JMS gateway is listening. The listener receives the message and the ESB converts it to an ESB-aware message and then passes it onto the ESB through the queue/quickstart_Business_Rules_Request_ESB queue.

Now it starts to get more interesting.

Remember that our rules rely on certain types of facts (which are JavaBeans) being available in working memory. Where do these facts come from? We'll create them out of that message.

Step 2 - Transform the Message into Beans

The problem is that we have to have a way to create those facts out of the information in the message. Luckily, one of the tasks that the JBossESB in the SOA Platform performs is "transformation." The next action in the quickstart's action pipeline uses the smooks processing engine (http://www.smooks.org/) and the JBossESB's out-of-the-box "SmooksAction" action to perform the transformation of the information in the message into facts:

1:  <action name="transform"
2:    class="org.jboss.soa.esb.smooks.SmooksAction">
3:      <property name="smooksConfig" value="/smooks-res.xml" />
4:      <property name="resultType" value="JAVA" />
5:  </action>

The source code for the facts (remember that these are JavaBeans and have getter and setter methods) is in these source files in the quickstart:

In: src/org/jboss/soa/esb/samples/quickstart/businessrules/dvdstore

Customer.java
OrderHeader.java
OrderItem.java

The quickstart uses the smooks-res.xml file to perform the transformation. Let's take a look at this file.

1:  <?xml version='1.0' encoding='UTF-8'?>
2:  <smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
3:             xmlns:jb="http://www.milyn.org/xsd/smooks/javabean-1.2.xsd">
4:
5:    <!-- Populate the OrderHeader -->
6:    <jb:bean beanId="orderHeader" class="org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.OrderHeader" createOnElement="order">
7:      <jb:value property="orderId"   data="Order/@orderId" />
8:      <jb:value property="orderDate"  data="Order/@orderDate" decoder="Calendar">
9:        <jb:decodeParam name="format">EEE MMM dd HH:mm:ss z yyyy</jb:decodeParam>
10:              <jb:decodeParam name="locale-language">en</jb:decodeParam>
11:        <jb:decodeParam name="locale-country">US</jb:decodeParam>
12:      </jb:value>
13:      <jb:value property="statusCode" data="Order/@statusCode" />
14:      <jb:value property="netAmount"  data="Order/@netAmount" />
15:      <jb:value property="totalAmount" data="Order/@totalAmount" />
16:      <jb:value property="tax"     data="Order/@tax" />
17:    </jb:bean>
18:
19:    <!-- Populate the Customer -->
20:    <jb:bean beanId="customer" class="org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.Customer" createOnElement="customer">
21:       <jb:value property="userName" data="customer/@userName" />
22:       <jb:value property="firstName" data="customer/@firstName" />
23:       <jb:value property="lastName" data="customer/@lastName" />
24:       <jb:value property="state"   data="customer/@state" />
25:    </jb:bean>
26:
27:    <!-- Populate the OrderItem list -->
28:    <jb:bean beanId="orderItemList" class="java.util.ArrayList" createOnElement="orderlines">
29:       <jb:wiring beanIdRef="orderItem" />
30:    </jb:bean>
31:
32:     <!-- Populate the OrderItem instance -->
33:     <jb:bean beanId="orderItem" class="org.jboss.soa.esb.samples.quickstart.businessrules.dvdstore.OrderItem" createOnElement="orderlines/orderline">
34:       <jb:value property="position" data="orderline/@position" />
35:       <jb:value property="quantity" data="orderline/@quantity" />
36:       <jb:value property="productId" data="orderline/product/@productId" />
37:       <jb:value property="title"   data="orderline/product/@title" />
38:       <jb:value property="price"   data="orderline/product/@price" />
39:    </jb:bean>
40:
41:   </smooks-resource-list>

That's right - it's using XPath (http://www.w3.org/TR/xpath/) to parse the information in the message into Customer (starting at line NN), OrderHeader (starting at line NN), and OrderItem.java (starting at line NN) JavaBeans and a java.util.ArrayList object that contains a list of the orders.

Step 3 - And Add the Beans Back into the Message

What happens next? Well, we have the original message, and some JavaBeans. But, remember that on the JBossESB in the SOA Platform, everything is either a message or a service. What we need is some way to get those JavaBeans back into the message. The way that we do this is with the next action in the action pipeline:

1:  <action name="map_order_components" class="org.jboss.soa.esb.actions.scripting.GroovyActionProcessor">
2:    <property name="script" value="/map_order_components.groovy" />
3:  </action>

GroovyActionProcessor is another of the SOA Platform's out-of-the-box actions. The groovy script referenced by the action takes the JavaBeans that we just created and adds them (note the use of the beanId's that we defined in the smooks-res.xml file) back into the message with property names that match the bean IDs. The script is very short - lines 3 and 4 add the orderHeard and customer to the message body:

1:    // Need to map down the orderHeader and customer beans onto the message
2:    // to make them available to the ObjectMapper...
3:    message.getBody().add("orderHeader", message.getBody().get().get("orderHeader"));
4:    message.getBody().add("customer", message.getBody().get().get("customer"));

Now we have the JavaBeans in the message. What happens next? The quickstart updates the customer status (it's set to a value of "0" in the JMS message that started the quickstart) with the "UpdateCustomerStatus" custom action:

1:  <!-- Update Customer Status -->
2:  <action name="updateCustomerStatus"
3:    class="org.jboss.soa.esb.samples.quickstart.businessrules.UpdateCustomerStatus">
4:      <property name="status" value="60"/>
5:  </action>

We'll set this to a value of 60 as the customer is in the platinum customer class. (Hint: For extra credit, try the quickstart with different status values.)

Here's the output in the log - this line is printed by the org.jboss.soa.esb.samples.quickstart.businessrules.UpdateCustomerStatus custom action:

1:  21:42:15,793 INFO [STDOUT] { Updated customer status to 60}

Step 4 - Process the Message with the BusinessRulesProcessor

OK - now we can see the BusinessRulesProcessor execute in the next action:

1:    <!-- Use the BRP to calculate the order priority -->
2:     <action
3:      class="org.jboss.soa.esb.actions.BusinessRulesProcessor"
4:        name="BRP">
5:          <property name="ruleSet"
6:            value="MyBusinessRules.drl" />
7:          <property name="ruleReload" value="true" />
8:          <property name="object-paths">
9:            <object-path esb="body.orderHeader" />
10:           <object-path esb="body.customer" />
11:          </property>
12:    </action>
13:
14:    <action name="reviewMessage1"
15:      class="org.jboss.soa.esb.samples.quickstart.businessrules.ReviewMessage">
16:        <property name="stuff" value="After Order Priority"/>
17:    </action>

Line 1 - Comments are always good things! ;-)
Line 2 - Here's the start of the action that makes use of the BusinessRulesProcessor.
Line 3 - And here's the reference to the BusinessRulesProcessor class.
Lines 5-6 - And, here's the reference to the Rules file that we want to execute.
Line 7 - This property causes the Rule to be if the file changes.
Lines 8-11 - And, here are the objects that we added to the message - remember the map_order_components.groovy file?

So, what just happened? We made the orderHeader and customer objects available to the Rules defined in MyBusinessRules.drl and executed the Rules. The net effect of this is that the priority of the order defined in the message should have changed. The next action in the action pipeline writes this to the log:

1:  21:42:15,931 INFO [STDOUT] Platinum Customer - High Priority
2:  21:42:15,932 INFO [STDOUT] Customer Status: 60
3:  21:42:15,932 INFO [STDOUT] Order Total: 64.92

The first three lines are printed by the rules. Remember how the rules all included System.out.prinln statements? And how the logging rule that prints the customer status and order total has a lower value salience property than the rules that set and print the customer priority? That explains the order in which the statements are printed to the log.

And then these lines are printed by the org.jboss.soa.esb.samples.quickstart.businessrules.ReviewMessage cusom action:

1:  21:42:15,932 INFO [STDOUT] { ================ After Order Priority
2:  21:42:15,933 INFO [STDOUT] Customer: user1,Harry,Fletcher,SD,60
3:  21:42:15,933 INFO [STDOUT] Order Priority: 3
4:  21:42:15,933 INFO [STDOUT] Order Discount: 0.0
5:  21:42:15,933 INFO [STDOUT] Order Total: 64.92
6:  21:42:15,933 INFO [STDOUT] } ================ After Order Priority

The rule "Highest Priority Orders" defined in MyBusinessRules.drl has set the order priority to 3.

Next, the quickstart calls the BusinessRulesProcessor again, this time to determine the order discount:

1:   <!-- Use the BRP to calculate the order discount -->
2:   <action
3:     class="org.jboss.soa.esb.actions.BusinessRulesProcessor"
4:       name="BRP2">
5:       <property name="ruleSet"
6:         value="MyBusinessRulesDiscount.drl" />
7:       <property name="ruleReload" value="true" />
8:       <property name="object-paths">
9:         <object-path esb="body.orderHeader" />
10:         <object-path esb="body.customer" />
11:       </property>
12:    </action>
13:
14:    <action name="reviewMessage2"
15:      class="org.jboss.soa.esb.samples.quickstart.businessrules.ReviewMessage">
16:      <property name="stuff" value="After Order Discount"/>
17:    </action>

Line 6 - The rule "Customer Platinum Status" defined in the MyBusinessRulesDiscount.drl file is executed as our customer has both the highest priority as well as the status value of 60. That rule and the logging rule print out these statements to the log in this order:

1:  21:42:16,062 INFO [STDOUT] Platinum Customer - High Priority - Higher discount
2:  21:42:16,063 INFO [STDOUT] Customer Status: 60
3:  21:42:16,063 INFO [STDOUT] Order Total: 64.92

Next, the org.jboss.soa.esb.samples.quickstart.businessrules.ReviewMessage custom action is executed to print these statements to the log:

1:  21:42:16,063 INFO [STDOUT] { ================ After Order Discount
2:  21:42:16,063 INFO [STDOUT] Customer: user1,Harry,Fletcher,SD,60
3:  21:42:16,064 INFO [STDOUT] Order Priority: 3
4:  21:42:16,064 INFO [STDOUT] Order Discount: 8.5
5:  21:42:16,064 INFO [STDOUT] Order Total: 64.92
6:  21:42:16,064 INFO [STDOUT] } ================ After Order Discount

And there we see our generous 8.5% discount.

Next, the quickstart prints the entire message to the log with this out of the box action:

1:  <action name="sout" class="org.jboss.soa.esb.actions.SystemPrintln" />

And here's how it appears in the server.log:

1:  21:42:16,064 INFO [STDOUT] Message structure:
2:  21:42:16,064 INFO [STDOUT] [{orderHeader=1, java.util.GregorianCalendar[time=1163616328000,areFieldsSet=true,areAllFieldsSet=false,lenient=true,zone=sun.util.calendar.ZoneInfo[id="US/Eastern",offset=-18000000,dstSavings=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=US/Eastern,offset=-18000000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=?,YEAR=2006,MONTH=10,WEEK_OF_YEAR=?,WEEK_OF_MONTH=?,DAY_OF_MONTH=15,DAY_OF_YEAR=?,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=?,AM_PM=?,HOUR=?,HOUR_OF_DAY=13,MINUTE=45,SECOND=28,MILLISECOND=?,ZONE_OFFSET=-18000000,DST_OFFSET=0], 0, 59.97, 64.92, 4.95, , orderItemList=[1,1,364,The 40-Year-Old Virgin ,29.98, 2,1,299,Pulp Fiction,29.99], orderItem=2,1,299,Pulp Fiction,29.99, customer=user1,Harry,Fletcher,SD,60}].

That's a little hard to read. Let's take out the order date/time so that we can better see the customer and orderHeader Beans that we added to the message:

1:  {
2:  orderHeader=1, 0, 59.97, 64.92, 4.95,
3:  orderItemList=[1,1,364,The 40-Year-Old Virgin ,29.98, 2,1,299,Pulp Fiction,29.99],
4:  orderItem=2,1,299,Pulp Fiction,29.99,
5:  customer=user1,Harry,Fletcher,SD,60
6:  }

Step 5 - Route the Message to its Destination Based on its Content

At this point, the quickstart has finished using the BusinessRulesProcessor, but it's not yet done using Rules. Remember how I said that on the JBossESB in the SOA Platform everything is either a message or a service? Well, one of the main functions performed by the ESB is to route messages to the correct service. These routes can be static, or they can be dynamic, based on the content of a message. Here's where content based routing with Rules comes in. The routing rules are defined in the MyRoutingRules.drl file. Remember how the rules in this file designated the "destinations" of the messages? This next action uses invokes the org.jboss.soa.esb.actions.ContentBasedRouter class to route the messages to their intended destinations.

1:  1  <!-- Use the CBR to route the "scored" order to the appropriate service team -->
2:  2  <action
3:  3    class="org.jboss.soa.esb.actions.ContentBasedRouter"
4:  4     name="ContentBasedRouter">
5:  5       <property name="ruleSet" value="MyRoutingRules.drl" />
6:  6       <property name="ruleReload" value="true" />
7:  7       <property name="destinations">
8:  8         <route-to
9:  9           destination-name="SuperSpecialCustomerService"
10:  10          service-category="ConciergeManager" service-name="ConciergeService" />
11:  11       <route-to
12:  12         destination-name="SpecialCustomerService"
13:  13         service-category="DistributionManager" service-name="DistributionService" />
14:  14       <route-to
15:  15         destination-name="RegularCustomerService"
16:  16         service-category="BasicShipping" service-name="ShipperService" />
17:  17       </property>
18:  18       <property name="object-paths">
19:  19         <object-path esb="body.orderHeader" />
20:  20         <object-path esb="body.customer" />
21:  21       </property>
22:  22 </action>

Lines 107-109, 110-112, and 113-115 define the routes. Since our message now has the highest priority, it is routed to the ConciergeService.

Here's what the log shows us. Again, the first line is printed by the rule:

1:  21:42:16,210 INFO [STDOUT] HIGHEST PRIORITY

And the remaining lines are printed by the org.jboss.soa.esb.samples.quickstart.businessrules.ReviewMessage custom action as invoked by the ConciergeService.

1:  21:42:16,309 INFO [STDOUT] { ================ Concierge
2:  21:42:16,309 INFO [STDOUT] Customer: user1,Harry,Fletcher,SD,60
3:  21:42:16,309 INFO [STDOUT] Order Priority: 3
4:  21:42:16,309 INFO [STDOUT] Order Discount: 8.5
5:  21:42:16,310 INFO [STDOUT] Order Total: 64.92
6:  21:42:16,310 INFO [STDOUT] } ================ Concierge

Well, the customer's order was delivered to the very posh ConciergeService, and the quickstart's execution is complete.

Closing Thoughts

OK, let's review what happened. The quickstart defines multiple business rules to examine and modify a message as it is processed by actions executed by ESB services, then it routes that message to the correct destination service. The rules are maintained in .drl files, separate from the services' custom action code, which makes it easier to maintain them.

And, notice what the quickstart did not have to do - the rules were executed and the content based routing was performed through out of the box actions provided by the SOA Platform's JBossESB. This made it possible for the quickstart to concentrate on the Rules business logic.

Acknowledgements

As always, I want to thank the JBoss SOA Platform team and community (especially Kevin Conner and Mark Proctor for their timely review input for this blog post).

References

JBoss SOA Platform (http://www.jboss.com/products/platforms/soa/)
JBoss BRMS Platform (http://www.jboss.com/products/platforms/brms/)
JBossESB Project (http://www.jboss.org/jbossesb)
JBoss Drools Project (http://www.jboss.org/drools)
JBoss Smooks Project (http://www.smooks.org)
JBoss Drools Business Rules by Paul Browne (published by Packt Publishing) (https://www.packtpub.com/jboss-drools-business-rules/book)

The Good, the Bad, and the Vague

2010-06-14T19:01:00.000-07:00

That last post got me thinking in concrete terms about...ambiguity.

Most of us experience being on the receiving end of ambiguity at an early age. It happens on our birthdays when we ask our parents for a dirt bike or air rifle. Some parents just say "no," but, the truly creative parents make use of the power of ambiguity and say "we'll see." Your parents are practicing intentional ambiguity.

Unintentional ambiguity, however, can be just as powerful, and, in engineering disciplines, can have serious consequences. When you're dealing with a physical medium such as glass or steel, you need to have an unambiguous understanding of the medium's limitations. For example, when you're building a bridge, you need to know exactly how much stress the steel handle. A design specification that states that the steel "might be at least as strong as this type of bridge generally requires" could lead to problems like this:

In the previous post to this blog, I discussed a couple of instances where unclear communications in source code and test reports had unpleasant side effects. How can you avoid situations like this, or problems caused by ambiguous product specification, requirement, or design documents? By including ambiguity reviews into your technical reviews.

I'd like to take credit for the idea of ambiguity reviews, but I can't. A formalized process for ambiguity reviews was defined and documented by Richard Bender here: http://benderrbt.com/Ambiguityprocess.pdf

The way it works is that it converts your technical specification reviews into a 2-phase process; first a review for ambiguity, and once any problems are resolved, your technical review. What sorts of things should you look for in an ambiguity review?

Adjectives such as: frequent or infrequent, many or few, normal or unusual
Adverbs such as: in general, not quite, hardly ever (remember the bridge!)
Verbs such as: maximize, efficiently, seamlessly (these sound like "resume words," don't they ;-)

In short, descriptions should be in measurable terms. Just like constants in an equation.

It's common to think of software testing problems in algebraic terms in that when you are writing a test or chasing a bug, you're trying to resolve variables to be able to reach a conclusion. If you can remove ambiguity from the information on which you base your tests, you can reduce the number of those variables. But, don't stop there! You should subject your own test plans to same ambiguity reviews so that your tests are just as precise as the product definition.

References:

http://benderrbt.com

(Special thanks to Mike for inspiring this!)

Barely Adequate Reports and the NADA Packet - The Value of Clear Communications in Software Testing

2010-06-04T05:41:00.000-07:00

One of the most important non-technical skills for a software test engineer is the ability to communicate clearly. In software testing, you are frequently confronted with situations where you must describe complicated tests, system configurations, and sequences of events. And, you sometimes have to do this during moments of stress during project development and test cycles, where the last thing that anyone wants to hear about is vague descriptions of yet another "interesting" bug.

I talking to former co-workers of former companies recently, I remembered a couple of instances from years ago where I saw, or was guilty of, less than clear communications. Each of these instances presented a different type of "teachable moment."

The first one involved a problem with lack of substantive detail. I had submitted a test analysis report on the state of a product's quality where I said that it was "barely adequate" for release to beta. My boss then informed me that my report was "barely adequate" as it was more of an editorial than a factual report. ;-) The correct approach to have taken in the report would have been to compare the state of the product to its defined beta test requirements in clearly measurable terms and not in personal opinions.

The second one involved a problem with language and jargon. In reviewing some existing tests, I noticed a comment in the code that read "then NADA packets result." It took me a few puzzled minutes to realize that "NADA" was not an obscure 4 letter acronym, but the Spanish word for "nothing." Yes, it would have been a lot easier if the comments had been at least as clear as the code they were meant to augment.

So, to ensure that you're communicating in a better than "barely adequate" manner, stick to the facts, and keep things clear.

(Gracias Cheryl!)

JBoss Enterprise Application DZone Refcard!

2010-05-04T10:40:00.000-07:00

I am very proud to be a contributor to the new JBoss Enterprise Application DZone Refcard!

The refcard is here: http://refcardz.dzone.com/refcardz/getting-started-jboss?oid=hom22213

Great Brain-Storming Tool

2010-04-25T20:07:00.000-07:00

A co-worker recently showed me a great brainstorming tool - XMind (http://www.xmind.net/).

It's cool to work with as you can easily convert your thoughts to a graphical model. For example, here are some quick and semi-random thoughts about what it takes to be a software test engineer, all arranged into an organized map:

(Thanks Prabhat!)

Cross Posted!

2010-04-20T19:37:00.001-07:00

The previous post - on JMS Transactions has been re-posted to the JBoss SOA Platform blog:

http://jboss-soa-p.blogspot.com/2010/04/jms-transactions-and-soa-platform.html

And DZone:

http://soa.dzone.com/articles/jms-transactions-soa-platform

JMS Transactions and the SOA Platform

2010-04-11T19:19:00.001-07:00

The Server's Down - And What's Worse, I Lost my Money!

Regardless of how carefully you plan, it's inevitable that the software you build will encounter a situation or chain of events that you did not anticipate, and experience some type of failure. (Trust me on this. I work in software testing and spend most of my waking hours either causing software failures, or debugging them.) Since it is impossible to avoid every failure, the ability to recover from failures is a crucial part of every software system design.

A classic type of failure involves moving money between bank accounts. It's a 2-phase set of actions. First, money is withdrawn from one account and, second, the money is deposited into a second account. This sounds simple, right? Well, suppose that the the system encounters a failure after it has withdrawn the money from the first account, but before it has had a chance to deposit into the second account? What happens to the money? How can you avoid losing this money? The answer is that you enclose these two actions into a transaction.

Transactions and ACID Properties

What exactly is a "transaction?" Like a lot of things in software, it's a buzzword that is overused. But beyond that, a transaction is a logical grouping of actions into a single larger action, where this larger action is performed, or not performed, based on an all-or-nothing evaluation of the results of the individual actions[1]. In the case of the transfer of money example we just mentioned, wrapping the two transfers into a transaction would ensure that if either transfer failed, your money would not be lost, and both accounts would be restored to their original condition.

Transactions are frequently described in terms of having an "ACID"[2] set of properties:

Atomicity - When we talk about wanting a transaction to be "atomic" what we mean is that we want the transaction, which will be made up of a set of individual actions, to be treated as a single atomic unit. If any of these actions fails, such as one of the bank account transfers in our example, then the entire transaction is canceled. (The term used for this is "rolled back.") A transaction is an all-or-nothing construct. All the actions enclosed by the transaction must be completed successfully before the transaction itself is completed (or "committed.")
Consistency - Transactions don't leave the data that it works on in a partially finished state. If a transaction rolls back, the data or database in question is returned to the (consistent) state that it was in before the transaction started.
Isolation - In the context of a transaction, "isolation" refers to how any conditions or state created by a transaction are not visible to any other transaction. For example, if our bank account transfer was executed in a transaction, then while it was executing, any other transactions or operations would not be able to see any of results of the transaction until the entire transaction had completed.
Durability - One of the dangers of the bank account transfer example, if it were to be attempted outside of a transaction, is that if something went wrong, data (or even worse, money!) would be lost. Because a transaction guarantees the consistency of the data, that data is durable. If the transaction rolls back, the original data is restored.

Transactions and the SOA Platform's JBossESB

OK, this is all interesting, but it sounds like something better suited to databases than Service Oriented Architecture. How does a transactional model apply to the JBossESB in the SOA Platform where everything is either a message or a service?

Here's how: some transports supported by the Platform (InVM[3] and JMS) support a transactional delivery of messages. The actual delivery of that message will not happen until the transaction is committed. How can you cause a rollback of a transaction in an action pipeline to occur? By configuring the application and its services to include the transaction and then by raising a RuntimeException in the action pipeline. The best way to explain and illustrate how this is with one of the SOA Platform's "quickstart" example programs. Let's take a look.

The Quickstart

One of the great features of the SOA Platform is its extensive, and always growing, set of "quickstart" programs. These programs illustrate various features supported by the Platform and serve as a great resource for writing your own applications. For our example, we'll look at the aptly named "jms_transacted" quickstart.

This quickstart illustrates using the JMS transport with the JBossESB in the SOA Platform to handle transactions. The quickstart also shows how message redelivery can be performed with the JMS transport. Before we examine the quickstart's code, configuration files, and actual output, it's important that we make note of the quickstart's use of JCA (Java Connector Architecture) [4]. JCA provides a standardized way to connect to JMS providers. For the JBossESB in the SOA Platform, jms-jca-providers provide transaction support for the action pipeline by enclosing actions in a JTA [5] transaction. These transactions ensure that messages are handled within an enclosing transaction. If something goes wrong in the transaction, the messages are put back into the JMS queues to be reprocessed. We'll see this in action when we run the quickstart.

The sequence of actions that the quickstart performs is:

Like many of the SOA Platform's quickstarts, things begin by having a JMS message routed to a service to start the action pipeline.
The first time that the pipeline is entered, a row is inserted into the quickstart's HSQLDB database. Note that while the SOA Platform supports multiple databases such as MySQL, Oracle, PostgreSQL and others, in production environments, it includes a fully functional HSQLDB database for demonstration purposes.
Then, the org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction class is called to raise an exception. This action class is configured to raise the exception (5) times, so as to force a rolback of the transaction that encloses the writes to the database. Each time that the exception is raised, the JCA adapter rolls back the transaction. The exception is also propagated to the JCA adapter and written error messages to the log.
After the configured limit of (5) rollbacks is reached, the org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction class completes and the transaction is committed by the SOA Platform's JBossESB's org/jboss/soa/esb/common/JBossESBTransactionService class so that the net result is only one row written to the database.

Now, we'll take a detailed look at how this transaction is configured, and just what happens as the quickstart runs.

The Quickstart - In Detail

The transaction in quickstart includes both JMS messaging and writing to a database. Let's start by examining the connection to the HSQLDB database that the quickstart uses. First, we need to create a datasource definition to connect to the database.

In file: quickstart-ds.xml

    <?xml version="1.0" encoding="UTF-8"?>
<datasources>
<local-tx-datasource>
   <jndi-name>JmsTransactedDB</jndi-name>
   <connection-url>jdbc:hsqldb:hsql://localhost:1706</connection-url>
   <driver-class>org.hsqldb.jdbcDriver</driver-class>
   <user-name>sa</user-name>
   <password></password>
   <min-pool-size>5</min-pool-size>
   <max-pool-size>20</max-pool-size>
   <idle-timeout-minutes>0</idle-timeout-minutes>
   <depends>jboss:service=JmsTransactedDB</depends>
   <prepared-statement-cache-size>32</prepared-statement-cache-size>
</local-tx-datasource>

<mbean code="org.jboss.internal.soa.esb.dependencies.HypersonicDatabase"
 name="jboss:service=JmsTransactedDB">
 <attribute name="Port">1706</attribute>
 <attribute name="BindAddress">localhost</attribute>
 <attribute name="Database">JmsTransactedDB</attribute>
 <attribute name="Silent">true</attribute>
 <attribute name="Trace">false</attribute>
 <attribute name="No_system_exit">true</attribute>
 <attribute name="DataDir">${jboss.server.data.dir}</attribute>
</mbean>
</datasources>

Some items to make note of here are:

Line 3 - We're defining a local-tx-datasource as we will want a JCA connection with transaction support[6].
Line 4 - We're use this JNDI name to reference the datasource.
Lines 5-13 - DB driver class, connection, URL, username/password, etc. for the database connection
Line 12 - MBean interface for running Hypersonic in the same VM with JBoss - note that the service name and the JNDI name as we'll refer to them later.

OK, we have a datasource that we can use to get to the HSQLDB database, and get us there in a mode that supports transactions. What's next? We have to ensure that our database actually exists before we try to use it! Luckily, the SOA Platform has a class that can initialize our database when the .esb application is deployed.

In file: jbossesb-service.xml

    <?xml version="1.0" encoding="UTF-8"?>

<server>
<mbean code="org.jboss.internal.soa.esb.dependencies.DatabaseInitializer"
    name="jboss.esb:service=JmsTransactedDatabaseInitializer">
   <attribute name="Datasource">java:/JmsTransactedDB</attribute>
   <attribute name="ExistsSql">select * from jms_transacted_table</attribute>
   <attribute name="SqlFiles">
     hsqldb/create.sql
   </attribute>
  <depends>jboss.jca:name=JmsTransactedDB,service=DataSourceBinding</depends>
</mbean>
</server>

There are (3) interesting things in this file:

Line 4 - This MBean creates the database on startup. How does it connect to the database? See line 6.
Line 6 - Here's a reference to our datasource. Note the JNDI name that we observed in the datasource definition file. How does the DatabaseInitializer know what the database schema look like? See line 9.
Line 9 - Here's a reference to the quickstart's src/hsqldb/create.sql file. This file create a very simple table in our database for the quickstart to use:

In file: src/hsqldb/create.sql

create table jms_transacted_table
(
unique_id INTEGER GENERATED BY DEFAULT AS IDENTITY(START WITH 1) NOT NULL PRIMARY KEY,
data_column VARCHAR(255) NOT NULL
);

What else does the quickstart's database need? The service that creates the database (JmsTransactedDatabaseInitializer) has to be deployed. This is handled in the deployment.xml file (as are the JBoss Messaging queues that the quickstart will use):

<jbossesb-deployment>
<depends>jboss.esb.quickstart.destination:service=Queue,name=quickstart_jms_transacted_Request_esb</depends>
<depends>jboss.esb.quickstart.destination:service=Queue,name=quickstart_jms_transacted_Request_gw</depends>
<depends>jboss.esb:service=JmsTransactedDatabaseInitializer</depends>
</jbossesb-deployment>

And finally, the jbossesb-service.xml quickstart-ds.xml files have to be included in the quickstart's .esb application archive file. This is handled by the "additional.deploys" property in the quickstart's build.xml file and the ../conf/base-build.xml file that is used by all the quickstarts.

cat -n build.xml | grep quick


<property name="additional.deploys" value="jbossesb-service.xml quickstart-ds.xml" />

OK, that takes care of the database. Now, let's look at how we configure the quickstart to take advantage of the transaction support on the JBossESB in the SOA Platform. As always, the place to begin is the jboss-esb.xml file.

In file: jboss-esb.xml

    <?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">

     <providers>
             <jms-jca-provider name="JBossMessaging" connection-factory="XAConnectionFactory">

                     <jms-bus busid="quickstartGwChannel">
                             <jms-message-filter
                                     dest-type="QUEUE"
                                    dest-name="queue/quickstart_jms_transacted_Request_gw"
                                    transacted="true"
                    />
                    </jms-bus>
                    <jms-bus busid="quickstartEsbChannel">
                            <jms-message-filter
                                    dest-type="QUEUE"
                                    dest-name="queue/quickstart_jms_transacted_Request_esb"
                                    transacted="true"
                    />
                    </jms-bus>
                    <activation-config>
                            <!-- The maximum number of times a message is redelivered before it is sent to the DLQ -->
                            <property name="dLQMaxResent" value="5"/>
                    </activation-config>

            </jms-jca-provider>
    </providers>

    <services>
            <service
                    category="JMSSecuredESB"
                    name="SimpleListener"
                    description="JMS Secured quickstart sample">
                    <listeners>
                            <jms-listener name="JMS-Gateway"
                                    busidref="quickstartGwChannel"
                                    is-gateway="true"/>
                            <jms-listener name="jmssecured"
                                    busidref="quickstartEsbChannel"/>
                    </listeners>
                    <actions mep="OneWay">

                            <action name="printMessage" class="org.jboss.soa.esb.actions.SystemPrintln">
                                    <property name="message" value="JMS Transacted Quickstart entered. Message body"/>
                                    <property name="printfull" value="false"/>
                            </action>

                            <action name="insertDBAction" class="org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction">
                                    <property name="datasource-name" value="java:JmsTransactedDB"/>
                                    <property name="db-insert-sql" value="insert into jms_transacted_table(data_column) values(?)"/>
                            </action>

                            <!--
                                    Will throw an Exception for the configured number of rollbacks. This should trigger the transaction to be
                                    rolledback and the message placed back onto the JMS queue.
                            -->
                            <action name="throwExceptionAction" class="org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction">
                                    <property name="rollbacks" value="5"/> <!-- if greater than dLQMaxResent then message goes to the JMS DLQ -->
                            </action>

                            <action name="printMessageDone" class="org.jboss.soa.esb.actions.SystemPrintln">
                                    <property name="message" value="JMS Transacted Quickstart processed successfully. Message body"/>
                                    <property name="printfull" value="false"/>
                            </action>

                            <!-- The next action is for Continuous Integration testing -->
                            <action name="testStore" class="org.jboss.soa.esb.actions.StoreMessageToFile"/>
                    </actions>
            </service>
    </services>

</jbossesb>

Line 2 - Before we go any further, we should look at the XSD (XML schema definition) for jboss-esb.xml files. The schema definition is available here: http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd. If you're going to be working with the SOA Platform, it's a good idea to become familiar with this XSD as all the elements that you work with in jboss-esb.xml are defined there. The two main elements are of course, providers and services. Let's look at the providers first. Two types of providers are supported. Schedule providers define schedule driven listeners that pull messages from queues at (you guessed it) scheduled time periods. In contrast, bus providers such as the jms-jca-providers that we're using in this quickstart, are associated with listeners that have messages pushed to them. [7]
Line 5 - Here's the start of our jms-jca-provider. Note that the XAConnectionFactory provides support for distributed transactions
Lines 7-13 - This is a reference to the channel and JMS queue that will be used when a JMS message that is sent to a JMS gateway to initiate the quickstart. Let's stop for a second and talk about the forms in which data is moved onto and across the JBossESB in the SOA Platform. One of the primary functions performed by JBossESB is the routing of messages between services. (I said it before, and I'll say it again; remember that, on an ESB, everything is either a message or a service.) The messages that JBoss ESB handles conform to org.jboss.soa.esb.message.Message. Service endpoints that can process messages in this format are described as being "ESB-aware." This is all well and good for services and applications that were designed and written with this message format in mind, but what about other services, including legacy applications, that communicate through other non-ESB aware data formats? JBossESB handles connecting services that communicate in non-ESB aware message formats through its gateway listeners. These listeners route incoming messages from outside the ESB to ESB services. The ESB's set of gateway listeners (generally referred to as "gateways") support a variety of message formats such as files, FTP, and JMS. These gateways accept messages in a non-ESB aware format onto the ESB and then convert them (actually, they wrap the message payload) into ESB-aware messages before sending them to their service's action pipeline.
Line 11 - Remember, we want the service to process messages within a transaction.
Lines 14-20 - Gateways are intended to enable the JBossESB to communicate to external data sources. The gateways themselves don't move data across the ESB. Accordingly, for each Gateway channel that we define, we have to define a corresponding ESB-aware channel to be used to route messages within the ESB.
Line 21 - The activation-config defines how we link to the JCA. In the case of this quickstart, the interesting property is: dLQMaxResent. As the comment indicates, the “DLQMaxResent” controls how many times the JCA adapter resends the message before it is sent to the dead letter queue. This is a JBossESB service that handles messages for transports when the messages cannot be delivered. (The JMS transport actually has its own dead letter queue as JBoss Messaging is shipped with a set of pre-configured destinations defined in destinations-service.xml including: <mbean code="org.jboss.jms.server.destination.Queue" name="jboss.messaging.destination:service=Queue,name=DLQ") In this quickstart, the JCA adapter will try to send the message 5 times.

After our providers are defined, we can define our services.

A service is where thing happen. Each service definition includes a set of listeners and a sequential set of actions referred to as an "action pipeline." The listeners "listen" for incoming messages for the service and route those messages to the action pipeline.

After the service name and category are defined on lines 31 and 32, we define the service's listeners:

Lines 35-37 - Here's the JMS gateway listener definition. Notice how it references the bus ID of the channel that we defined in earlier in the provider definition. Also, notice how it's "is-gateway" property is set to true to indicate that this listener is a gateway.
Lines 38-39 - And here is the ESB-aware listener that corresponds to the gateway listener.

So much for the listeners, now let's look at the actions.

Line 41 - This is the start of the definitions of the sequence of actions that we referred to as the action pipeline. The message exchange pattern (or "mep") indicates that the messages are not being exchanged in a request/response pattern, but rather are being sent in only one direction. (In contrast, web services exposed on the ESB would follow a synchronous request/response pattern.)
Lines 43-46 - This action invokes the simplest of the JBossESB's many useful out-of-the-box actions (http://soa.dzone.com/articles/works-great-right-out-box) to write some messages to the server log.
Lines 48-51 - Here's where things start to get interesting. This custom action writes a row to our database table. Remember the "java:JmsTransactedDBdatasource" that we defined? Here's where gets used.
Line 57 - Remember how we talked about how to cause a rollback of a transaction in the action pipeline to occur? We configured the .esb to include a transaction, and now we raise a RuntimeException in the action pipeline. The action defined here raises an exception of a type that we define in the org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction class. The ActionProcessingPipeline will catch the exception and pass it up the stack to its caller, the JMS/JCA adapter. The Adapter then causes the transaction to rollback and the message is redelivered.
Line 58 - When we defined the jms-jca-provider's activation-config, we specified a dLQMaxResent value of 5 to indicate that if the transaction was rolled back more than 5 times, the message processed in the transaction should be sent to the JMS dead letter queue. In line 58, we also set the value of the rollbacks property equal to 5, to ensure that the transaction can complete after 5 rollbacks, and the message is not sent to the dead letter queue service. As the comment helpfully explains, setting the rollbacks property to be greater than dLQMaxResent will send the message to the dead letter queue service.
Lines 61-64 This action is only reached after the rollbacks have been performed and the transaction is able to complete. The action writes the message body to the server log.

Yogi Berra once said that "In theory there is no difference between theory and practice. In practice there is."[8] OK. Enough theory, let's run the quickstart and watch what happens.

After the SOA-P server is started, the quickstart, is deployed with this command: ant deploy

When the quickstart is deployed, the following is to the server.log:

2010-04-07 09:26:22,117 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (HDScanner) Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=JmsTransactedDB' to JNDI name 'java:JmsTransactedDB'
2010-04-07 09:26:22,157 INFO  [org.jboss.internal.soa.esb.dependencies.DatabaseInitializer] (HDScanner) Initializing java:/JmsTransactedDB from listed sql files
2010-04-07 09:26:22,326 INFO  [org.jboss.soa.esb.listeners.deployers.mc.EsbDeployment] (HDScanner) Starting ESB Deployment 'Quickstart_JMS_Transacted.esb'

Note how the JmsTransactedDB database is initialized by org.jboss.internal.soa.esb.dependencies.DatabaseInitializer class the when the quickstart is deployed.

Next, we run the quickstart with this command: ant runtest

When the ant runtest target is executed, the following chain reaction is started:

The org.jboss.soa.esb.samples.quickstart.jmstransacted.test.SendJMSMessage class sends a (non-ESB aware) JMS message to the quickstart_jms_transacted_Request_gw queue. The message is routed through that queue's corresponding ESB-aware queue (quickstart_jms_transacted_Request_esb) to the SimpleListener service and the action pipeline is initiated.

The first time that the action pipeline is started, the org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction class inserts a row into the database through the HSQLDB datasource. This results in the following messages being written to the server.log:

Then, the org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction runtime exception is thrown.

Since this exception is thrown from inside the action pipeline, the JMS/JCA adapter rolls back the enclosing transaction. The exception is propagated to the JMS/JCA adapter as the exception is passed up the stack from the action pipeline to its the caller (which is the adapter). The exception is caught by the JBossESB's org.jboss.soa.esb.listeners.message.ActionProcessingPipeline, and is propagated up the stack to the JMS/JCA adapter (which is the action pipeline's caller). The adapter causes the transaction to rollback. This results in the row being removed from the database and in the original message being redelivered. Remember what we said about a transaction being an "all or nothing" event?

The transaction/rollback sequence then repeats until the configured number of rollbacks are performed. You can see a counter (maintained by the org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction class) of the number of times the action is attempted.

The printMessage action prints the message to the log:

2010-04-07 09:26:35,340 INFO  [STDOUT] (WorkManager(2)-9) JMS Transacted Quickstart entered. Message body:
2010-04-07 09:26:35,340 INFO  [STDOUT] (WorkManager(2)-9) [Hello Transacted JMS World]].

And the insertDBAction action writes the row to the database - note the counter value of [1]:

2010-04-07 09:26:35,415 INFO  [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction] (WorkManager(2)-9) Successfully inserted [Hello Transacted JMS World] counter[1]] into jms_transacted_table

But wait! Then our exception is raised:

2010-04-07 09:26:35,488 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (WorkManager(2)-9) Unexpected error delivering message delegator->JBossMessage[5204569273565185]:PERSISTENT, deliveryId=0
java.lang.IllegalStateException: [Throwing Exception to trigger a transaction rollback]
at org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction.process(ThrowExceptionAction.java:75)

The JMS/JCA adapter rolls back the transaction, and the message is delivered again, and another row is written to the database. The counter is incremented to [2] this time and the exception is raised again.

2010-04-07 09:26:36,435 INFO  [STDOUT] (WorkManager(2)-10) JMS Transacted Quickstart entered. Message body:
2010-04-07 09:26:36,435 INFO  [STDOUT] (WorkManager(2)-10) [Hello Transacted JMS World]].
2010-04-07 09:26:36,438 INFO  [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction] (WorkManager(2)-10) Successfully inserted [Hello Transacted JMS World] counter[2]] into jms_transacted_table
2010-04-07 09:26:36,442 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (WorkManager(2)-10) Unexpected error delivering message delegator->JBossMessage[5204569273565185]:PERSISTENT, deliveryId=1
java.lang.IllegalStateException: [Throwing Exception to trigger a transaction rollback]
at org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction.process(ThrowExceptionAction.java:75)

When the rollback counter number is reached, the action pipeline completes and the transaction commits so that the row is finally written to the database. The following messages are written to the servler.log:

This sequence repeats (5) times until the counter is passed and the row is actually written to the database:

2010-04-07 09:26:40,489 INFO  [STDOUT] (WorkManager(2)-14) JMS Transacted Quickstart entered. Message body:
2010-04-07 09:26:40,489 INFO  [STDOUT] (WorkManager(2)-14) [Hello Transacted JMS World]].
2010-04-07 09:26:40,490 INFO  [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.DBInsertAction] (WorkManager(2)-14) Successfully inserted [Hello Transacted JMS World] counter[6]] into jms_transacted_table
2010-04-07 09:26:40,490 INFO  [STDOUT] (WorkManager(2)-14) JMS Transacted Quickstart processed successfully. Message body:
2010-04-07 09:26:40,490 INFO  [STDOUT] (WorkManager(2)-14) [Hello Transacted JMS World]].

Now, as I mentioned in the beginning of this post, I work as a software QE engineer. One of the basic tenents of software QE is to never trust anything. Let's take a closer look at what happens when the quickstart runs.

Server logs are goldmines for gathering debugging information. Let's restart the server, but this time set the server logging level to DEBUG:

./run.sh -Djboss.server.log.threshold=DEBUG

And then rerun the quickstart. The server.log will be much more verbose this time, but if you look closely, you can see the rollback counter in action:

grep  nr-of-rollbacks server.log
2010-04-07 23:14:19,776 DEBUG [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction] (WorkManager(2)-11) rollbackCounter [0], nr-of-rollbacks [5]
2010-04-07 23:14:20,804 DEBUG [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction] (WorkManager(2)-12) rollbackCounter [1], nr-of-rollbacks [5]
2010-04-07 23:14:21,819 DEBUG [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction] (WorkManager(2)-13) rollbackCounter [2], nr-of-rollbacks [5]
2010-04-07 23:14:22,833 DEBUG [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction] (WorkManager(2)-14) rollbackCounter [3], nr-of-rollbacks [5]
2010-04-07 23:14:23,848 DEBUG [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction] (WorkManager(2)-15) rollbackCounter [4], nr-of-rollbacks [5]
2010-04-07 23:14:24,858 DEBUG [org.jboss.soa.esb.samples.quickstart.jmstransacted.test.ThrowExceptionAction] (WorkManager(2)-16) rollbackCounter [5], nr-of-rollbacks [5]

Before we move on, let's look at one more thing and verify that only one row is actually written to the database. Now, the quickstart is very tidy and always cleans up after itself by removing rows from the database. Let's disable this cleanup by removing this target from the quickstart's build.xml file:

cat build.xml | grep truncate

And, let's start the server again, but with one additional command line option: sh ./run.sh -Djava.awt.headless=false

Why "headless=false?" After we run the quickstart again, the row that was written to the database should still be there. We'll look at it with the HSQL DB Manager service. This service requires the server to not be started in its default "headless" mode, as the service opens up a Java UI application. We invoke the DB Manager from the JMX console - see screenshot - the service tag is: database=jbossesb,service=Hypersonic.

Once the DM Manager application UI launches, we just connect to the "JmsTransactedDB" database - see screenshot - and execute a query to return all rows in the "jms_transacted_table" - see screenshot.

And, there is our one record!

Before we move on, it's worthwhile to note what we did NOT have to do to execute a transaction with the SOA Platform.

We did not have to write transaction specific code. The JBossESB in the Platform did the dirty work for us. All we had to do is to define the .esb application's configuration in its jboss-esb.xml file to include the transaction-relevant properties and reference a JMS/JCA adapter and the Platform did the rest. This is what makes middleware so useful. You could write the code to handle transactions all on your own, but wouldn't you rather concentrate on solving your own business problems instead of build infrastucture plumbing? [9]

Closing Thoughts

The value that using transactions adds to an application is pretty obvious (just keep your bank account in mind). The value added by middleware in implementing transactions should also be obvious. You could write your own code to handle atomicity, consistency, isolation and durability, but, this would not be an easy task, and it would also take away a lot of time from your being able to concentrate on solving your own business logic problems. The SOA Platform's support for transactions over the JBossESB enables you to provide a higher level of reliability and durablity to your service based applications.

References

[1] http://qconlondon.com/london-2010/file?path=/qcon-london-2010/slides/MarkLittle_TransactionsOverUsedOrJustMisunderstood.pdf

[2] Little, Mark, Maron, Jon, Pavlik, Greg. Transaction Processing: Design and Implementation, Upper Saddle RIver, new Jersey: Prentice Hall/Hewlett-Packard Professional Books, 2004.

[3] http://www.redhat.com/docs/en-US/JBoss_SOA_Platform/5.0.0/html-single/Programmers_Guide/index.html#sect-SOA_ESB_Programmers_Guide-What_is_a_Service-InVM_Transport

[4] http://java.sun.com/j2ee/connector/

[5] http://java.sun.com/javaee/technologies/jta/index.jsp

[6] http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Server_Configuration_Guide/4/html/Connectors_on_JBoss-Configuring_JDBC_DataSources.html

[7] http://www.redhat.com/docs/en-US/JBoss_SOA_Platform/5.0.0/html-single/Programmers_Guide/index.html#sect-SOA_ESB_Programmers_Guide-Configuration-Providers

[8] http://en.wikiquote.org/wiki/Yogi_Berra

[9] http://magazine.redhat.com/2008/03/11/what-is-middleware-in-plain-english-please

Acknowledgments

As always, I want to thank the JBoss SOA Platform team and community (especially Kevin Conner for his timely review input for this blog post).

The Value of a Fresh Set of Eyes

2010-03-24T16:56:00.000-07:00

The test failed.

I checked and rechecked the results. There were no errors or exceptions. The logs were all clean.

But, the test failed.

I rebooted the server and reran the test.

And the test still failed.

It's not an uncommon situation in software QE to encounter unexpected test failures, but this one was very puzzling. Everything seemed to be functioning correctly, bit the test always failed.

The test in question verified that data could traverse a complex series of actions in the software under test. In the course of being routed through the system, the data would be processed into multiple intermediate forms such as files and messages and then be sent between multiple subsystems over protocols including FTP and JMS.

"Divide and conquer" is an effective debugging technique, so I broke the test into pieces and then verified that each piece was functioning correctly. I checked to ensure that messages were correctly formatted and written to queues. I also checked to ensure that the intermediate work files, each created with a different file type, were created in the correct directory path on the various FTP servers.

And the test still failed.

Finally, after staring at this test for a few hours, I asked a co-worker to take a look at it. After about 5 minutes he laughed and said, "The files on the FTP are in the correct places and have the correct type extensions, but the files are empty." The problem was a matter of timing. The test was trying to read the files before any data was written to them.

In "The Cathedral and the Bazaar,"[1] Eric Raymond observed the principle that "many eyes make shallow bugs." He was referring to how in open source software, a program's code is freely available, and how this openness enables more people to identify and resolve a program's flaws.

It can be difficult to test your own code. After you spend time and invest effort in building a program you can take for granted how, and how well, it operates in certain circumstances. You can look at and simply not see its flaws.

It's a common, and very effective practice on software test teams to rotate assignments. This helps to ensure that no one develops a "stale" view of the software under test or begins to take any potential flaws in the software as an intrinsic part of the the code.

So, maybe we have another principle, "fresh eyes see things that you look right past."

References:

[1] http://www.catb.org/~esr/writings/homesteading/

Special thanks to Martin! Děkuji ti!

Two new posts (well, reposts) in DZone!

2010-02-25T12:09:00.001-08:00

And, here they are:

"From HTTP to the ESB - JBossESB 4.7's New Servlet based HTTP Gateway," http://architects.dzone.com/news/jbossesb-http-gateways

"Message Alerts in JBossESB 4.7," http://soa.dzone.com/news/messagealerts-jbossesb-47

Software Usability - Not Just in the Eye of the Beholder

2010-02-11T18:54:00.000-08:00

One of the most important parts of any software test definition is the set of "expected results" for the test. If you cannot define the results that you expect the test to generate, then you make the task of determining whether the test has passed or failed much more difficult. For functional tests, this task is fairly straightforward; does method ABC return a boolean true or false value?

For non-functional tests, the task is more complicated. Performance tests can be measured against a set of benchmarks and security tests can be built to ensure that the system under test does not exhibit any of a set of known vulnerabilities such as the OWASP "top 10" (http://www.owasp.org/index.php/OWASP_Top_Ten_Project).

But, what about usability tests? Is usability purely in the eye of the beholder?

I've always tended to work on the "server side" of software products. So, when I had to think about usability tests, I wanted to find some resources that would help me approach the test planning in an organized, systematic way.

I pretty quickly discovered the work of Jakob Nielsen. Dr. Nielsen has written extensively on the subject of usability and is probably the best known web usability consultants in the world. He has defined the five "quality components of usability goals" as learnability, efficiency, memorability, errors (error rate), satisfaction (http://www.useit.com/alertbox/20030825.html

In addition to his books and papers, he also maintains the http://www.useit.com website. This is a great place to learn about usability testing and to find information that can help you in defining specific usability tests.

When Speed (well, the lack of it) Kills

In my own case, I was concerned about how to quantify user interface performance or responsiveness. Dealing with an unresponsive user interface can quickly ruin a user's impression of your application, even if it is otherwise well designed and error free.

I heard an interesting story on the the radio a couple of days ago about a new launched social engineering site. The site designer was asked, "What was your best day so far?" He answered that their first day being on-line was the best day ("yesterday"). He was then asked, "What was your worst day?" He answered "today." The problem was that the site was quickly overloaded and was presenting users with such long delays that people were abandoning the site.

What I needed for my tests was a set of benchmarks to measure UI responsiveness, and do it in a systematic way, so that my test findings would be quantitative, and not based on my opinion of the UI as being "too slow."

What I found was Dr. Neilsen's benchmark definition - http://www.useit.com/papers/responsetime.html

.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning no special feedback is necessary except to display the result.
1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay.
10.0 seconds is the limit for keeping the user's attention focused on the dialog. for longer delays, the user will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.

It really helped me to be able to compare the responsiveness of the UI to a measureable baseline as I was able to avoid discussions such as:

User 1: This UI is slow.
User 2: How slow?
User 3: Too slow!

Choosing Kaizen Over Cheez Whiz

2009-12-27T19:05:00.000-08:00

The end of December is a good time to reflect on the year that is ending. You can look back on your accomplishments, your victories and defeats, and the wreckage (or just the abandonment) of your new years' resolutions from the previous year. This year, the end of the year has gotten me thinking of....software development and test processes.

One of my original goals in starting this blog was for it to as a resource for "new-to-it" software test engineers and managers. The general type of question that I want the blog to be able to answer is "now what do I do?" What I especially want is to be able suggest were specific actions that the reader can follow, instead providing only general, more abstract advice. These actions are related to three components of a successful software test team:

People - who to hire and when to hire them, how to organize them into a team, how to maximize the effectiveness of a team
Tools - which tools to use, and how to use them, and how to use them in combination, and
Processes - how to create a flexible, efficient, and repeatable set of procedures

The Monday Morning 9:00AM Question

One specific case where I want the blog to be useful involves the first steps that a new software test manager should perform. I want the blog to be able to help a person in a situation where it's the start of the first day after you've just been appointed software test manager. You have no team, no tools, and no processes in place. As you sit at your desk, you're not only asking yourself "what do I do?", you're asking yourself, "what do I do first?"

I addressed the "people" component aspect of this question in an earlier post to this blog where I discussed the characterIstics of the first person to add to a team. I'll write at a later date on the first tools that you'll want to start using and the best way to use them together. For today, I want to talk about the first process that you want to implement.

"We Own the Process"

For many software engineers (the author included), the word "process" can have a negative connotation, as it conjures up images of having to spend lots of your time doing everything other than writing code. I once worked for a manager who, when he announced at a department meeting that "we own the process" subsequently discovered that we had hidden several jaras of "cheez whiz" in his office. (In case anyone has not heard of "cheez whiz," it's a famous American delicacy. It cannot legally be called cheese as it is so heavily processed, instead, it is described as a "pasturized process cheese food."

That was an unfortunate and extreme incident as "process" really ought not to be a bad word. In order for a software test organization to be successful over the long term, it has to develop a process that can be relied on as a roadmap to lead the team through difficult problems. In fact, the very "process" (pardon the pun) of defining, examining and documenting your software development and test process can itself be a useful task in making your organization more effective. If you force yourself to write down the rules under which your organization operates, you will probably find ways to improve it. (But, this is a subject for a different blog post!) This process also has to be repeatable so that the team's success can be repeatable. And, most importantly of all, the process has to always be a means to an end, where that end is the creation of high quality software, on time and on budget.

To get back to subject of this post, which particular process should the team put into place first? There are several possible answers to this question:

Requirements traceability - In defining tests, you should be able to establish a relationship between the product's requirements and your tests tests. Any gaps represent holes in your test coverage.
Formal test planning - There's a place for ad hoc or exploratory testing[1] in any project, but it cannot represent the majority of a testing effort. You'll doomed to overlook some critical tests unless you impose some discipline on the process by compiling your test ideas into a plan. The goal of test planning is not to produce a document, but to create define the tests, ensure that you have proper test coverage, mitigate risks, etc. The document is almost a by-product to the act of collecting and defining the information in the document. Like I always tell people, "plan is also a verb."
Defect tracking - If you find a bug, but have no means to record it, is it still a bug? Obviously, you have to track and record bugs in a presistent data store.
Test results tracking - Likewise, if you run a test, but do not have the means to record the results for future analysis, and to compare the results with future test runs, then the test is a lot less useful than it could be.

Did you notice the common thread tying these processes together? They all involve maintaining information in a form that someone other than the original author or designer can use and in a form that is persisted in a database or document so that a wide audience can also use it. I like to use the phrase "institutional memory" to describe the output from these documentation and data recording tasks, as the information thus recorded becomes the memory or legacy of the testing.

Hey, what about test automation? Well, for the purposes of this discussion, I'm putting test automation in a separate category, sort of like breathing. Every team and every project has to invest in automation. Without automation, you can never achieve a sufficient level of test coverage, and the consistency of test execution and test results to ensure the quality of the software under test. In addition, you'll never be able to put into place an effective continuous integration process without a high degree of test automation. The degree to which automation is essential makes me treat this in a different class than other software test processes.

I think, however, if I had to advise someone as to the first process to set up for a new test team, I would choose "kaizen." [2] In other words, continuous improvement.

A systematic approach to continuous improvement can be effective in any complex human endeavor, whether it's software testing or formula 1 racing, or investment banking. You learn from past mistakes, modify your approach to tasks to incorporate those lessons learned, or, in Scrum terms, you continuously "inspect and adapt" [3].

But, what is it about software testing that makes continuous improvement especially applicable?

The Answer is: Bugs

"We deal in lead." [4]
- Steve McQueen in "The Magnificent Seven"

"We deal in bugs."
- Any software tester

The common denominator that ties all software testing together is that every test has the same ultimate goal; to find an as yet undiscovered bug.[5] When you practice software testing, you spend your time looking for, investigating, recreating, and trying to understand the root cause of software bugs. And, every bug presents an opportunity for some type of design or implementation improvement. What's more, the bugs that look for and miss, also present opportunities for improvement in test design and implementation.[6]

In software testing, you are always inspecting both the software product under test and adapting the test programs and test approach to the current state of that product. Your tests and test plans can never be static, they always have to adapt, and be improved, to meet the current conditions. And, it's important that this improvement be continuous, and not only be a task that you think about at the end of a project. In terms of a software product's development and testing lifecycle, waiting days or weeks to make a change to a test or adapt your test plan to deal with an unexpected set of new bugs.

So, how (specifically) can you implement a process of continuous improvement? Some ways are to:

Turn every new bug into a new test - This one is easy, whenever you find a new bug, write an automated regression test for it. This will ensure that if the bug is ever re-introduced, you'll be able to catch it.
Mine the customer support calls for new test ideas, and for missing tests - In the course of preparing your test plan, you should collect input from other product team members. You should also review the problems that actual customers are having. You may find that they are using the product in unexpected ways. I once saw a problem where a telephone voicemail system offered customers a feature where they could have the system call them back. People started to use this as a wakeup service, but the call back processes always ran at lower priority level than incoming call processes. What happened? A lot of people received some very late wakeup calls. ;-)
Regularly re-review existing plans and tests - Like I said earlier, the plans should be dynamic, not static.

Closing Thoughts

And, remember, "there is always something left to improve"[7] - No matter how successful your testing is, there will be bugs that you missed. The complexity of software products, coupled with the flexible (or is it "brittle"?) nature of software as a medium, means that changes will be made and bugs will be introduced. So, if you think that that your test plan and tests are "complete," think again. The surest way to fall behind is by standing still!

References

[1] http://swqetesting.blogspot.com/2007/11/in-praise-of-exploratory-testing.html

[2] http://en.wikipedia.org/wiki/Kaizen

[3] http://www.scrumalliance.org/pages/scrum_framework

[4] http://www.rateitall.com/i-854965-we-deal-in-lead-steve-mcqueen-the-magnificent-seven.aspx

[5] http://swqetesting.blogspot.com/2007/11/fundamental-1-goal-of-sw-test-is-to.html

[6] http://swqetesting.blogspot.com/2008_08_01_archive.html

[7] As much as I'd like to take credit for the line "there is always something left to improve," I can't. It's attributed to Ben Hogan (1912-1997), the great American golfer of the 1940's and 1950's. His life story is really fascinating. He was born into poverty, reportedly witnessed his father's suicide at age 7, survived a crippling automobile accident, and after decades of failure and constant practice, made himself into the best professional golfer of his time. He is also without a doubt the greatest self-taught athlete in history. No one, in any sport, has ever worked harder to constantly improve than Hogan. After he retired from competition, he founded a golf club manufacturing company. His products were known for their high level of quality as he would never sell anything of low quality with his name on it. In the early 1990's, I happened to have a broken golf club manufactured by the Hogan company. It was broken in such a way that it could not be repaired, so I sent a letter to the company asking to buy a replacement. I received a personal letter from Mr Hogan, asking me to send him the club. He wanted to examine it himself to understand if it was a manufacturing defect that could be improved on. (He later sent me a free replacement club too!)

(Special thanks to Jirka for the Kaizen inspiration!)

New post in the JBoss ESB blog

2009-12-22T19:02:00.000-08:00

New post in the JBoss ESB blog: http://jbossesb.blogspot.com/2009/12/from-http-to-esb-jboss-esb-47s-new.html

The last one for this year!

And still another DZone post!

2009-12-03T17:00:00.000-08:00

This is getting to be a habit...;-)

http://soa.dzone.com/news/impatient-start-jboss-riftsaw-0

Another post on the JBoss ESB blog!

2009-11-26T18:36:00.000-08:00

A cool new feature in the new JBoss ESB release:

http://jbossesb.blogspot.com/2009/11/messagealerts-in-jbossesb-47.html

And yet another post on DZone!

2009-11-18T09:41:00.001-08:00

http://soa.dzone.com/news/esb-bpel-continuing-riftsaw

Another DZone post!

2009-11-13T10:55:00.000-08:00

Riftsaw BPEL this time...

http://soa.dzone.com/news/bpel-esb-and-back-introduction

Better looking posts!

2009-11-13T06:49:00.000-08:00

FYI everyone, the code listings in future posts to my blog will look a lot better, thanks to:

Matt Ball: http://heisencoder.net/2009/01/adding-syntax-highlighting-to-blogger.html

Alex Gorbatchev: http://alexgorbatchev.com/wiki/SyntaxHighlighter

David Ward: http://codeformatter.blogspot.com/2009/06/about-code-formatter.html

And Kurt Stam: http://kurtstam.blogspot.com/2009/02/code-listings-on-blogger.html

Just take a look here for an example: http://swqetesting.blogspot.com/2009/10/shocked-by-disk-space-used-by-iphoto.html

Riftsaw - Open Source BPEL!

2009-10-28T19:03:00.000-07:00

I just started posting to another blog - this one is for the "Riftsaw" project. Riftsaw is JBoss's open source BPEL engine. Very cool stuff...

http://riftsaw.blogspot.com/2009/10/impatient-start-with-jboss-riftsaw-5.html

http://riftsaw.blogspot.com/2009/11/from-bpel-to-esb-and-back-introduction.html

http://riftsaw.blogspot.com/2009/11/from-esb-to-bpel-continuing-with.html

Shocked by the Disk Space Used by iPhoto - When QE Experience Comes in Handy

2009-10-18T18:37:00.000-07:00

I was backing up by iPhoto library today and was shocked to see that had recently grown in size by about 6GB. This increase in disk usage happened while I had added no more than (100) or so new pictures.

Huh?

At first, I assumed that some form of corruption had happened. On closer inspection, however, I found the true culprit:


du -h | grep G
6.0G    ./iPod Photo Cache
1.0G    ./Modified/2007
2.6G    ./Modified
2.0G    ./Originals/2007
1.6G    ./Originals/2008
1.2G    ./Originals/2009
6.8G    ./Originals
16G    .

Nothing had happened to the iPhoto library or database. The increase in disk space usage was due to my finally getting an iPod that could handle pictures. ;-)

But - there's a lesson here that all software QE engineers have learned. When you think you've found a new bug, you have to examine the software under test for any changes, AND you also have to examine the environment in which the software runs for changes.

I like to look at software testing as an exercise in algebra. Your task is to take a very complex equation and find its answer. In that equation, you have some variables to resolve. The trick some times is that the variables may not be so obvious at first! Even if they are 6GB in size...

Another DZone post!

2009-10-16T20:12:00.000-07:00

DZone just picked up the Byteman post - it's here:

http://java.dzone.com/news/fault-injection-testing-first?mz=3006-jboss

Thinking about Scrum and Base Running

2009-10-09T20:38:00.000-07:00

I attended an excellent class on Scrum[1] a couple of weeks ago. I've always been a bit of a skeptic of formal project management or development frameworks, but I have to say that I was impressed. Some elements of scrum are really either common sense, such as the need for transparency, and others, such as short development cycles that result in an always functioning system hold real promise.

The one aspect of scrum that I identified most with, however, was "discover and adapt." In order to be successful in testing software, you always have to operating in a discover and adapt mode. You can define a detailed test plan, and base that plan on the best information available at the time, but in the course of a software project test cycle, you have to constantly re-evaluate the current state of the software under test, and then adapt your future plans to match. You may start off by testing subsystem A, but then find that its testing is blocked, but you can continue to make progress by testing subsystem B.

One crucial part of your adapting to what you discover about the software under test, is that your priorities for testing will always change. You highest priority will always be to run tests that will find the most serious as yet undiscovered bug. But, the places in the code where that bug may be lurking can change as either the code changes, or your understanding of the code increases.

There's a great analogy for "discover and adapt" in sports. (And, no, it doesn't involve golf this time. ;-) It involves American baseball, and concerns the sometimes under appreciated skill of base running.

When you think about what it takes to be a great base runner, you might think that the only factor is speed. Speed is important, to be sure, but what is even more important is the base runner's judgment. He has to be able to adapt quickly to changing situations, and decide when to be aggressive and "take the extra base," and when to be more conservative. In short, he has to discover and adapt. How does he do this? He watches the ball and the opposing players, not the bases. The bases aren't changing, the position of the ball and those players is.

I found this wonderful passage about Joe DiMaggio in David Halberstam's book[2] "Summer of '49" that describes this:

...Stengel, his new manager, was equally impressed, and when DiMaggio was on base he would point to him as an example of the perfect base runner. "Look at him," Stengel would say as DiMaggio ran out a base hit, "he's always watching the ball. He isn't watching second base. He isn't watching third base. He knows they haven't been moved. He isn't watching the ground, because he knows they haven't built a canal or a swimming pool since he was last there. He's watching the ball and the outfielder, which is the one thing that is different on every play..."

It's like that in software too. You can't keep your eye on static plans, you have to keep your eye on the current (and changing) state of the project and the code, and always discover and adapt.

References

[1] http://en.wikipedia.org/wiki/Scrum_(development)

[2] Halberstam, David, "Summer of '49,", (Morrow: New York), 1989. http://www.amazon.com/Summer-49-P-S-David-Halberstam/dp/0060884266/ref=sr_1_1?ie=UTF8&s=books&qid=1255145843&sr=8-1

Fault Injection Testing - First Steps with JBoss Byteman

2009-10-09T08:51:00.000-07:00

Fault injection testing[1] is a very useful element of a comprehensive test strategy in that it enables you to concentrate on an area that can be difficult to test; the manner in which the application under test is able to handle exceptions.

It's always possible to perform exception testing in a black box mode, where you set up external conditions that will cause the application to fail, and then observe those application failures. Setting and automating (and reproducing) these such as these can, however, be time consuming. (And a pain in the neck, too!)

JBoss Byteman

I recently found a bytecode injection tool that makes it possible to automate fault injection tests. JBoss Byteman[2] is an open-source project that lets you write scripts in a Java-like syntax to insert events, exceptions, etc. into application code.

Byteman version 1.1.0 is available for download from: http://www.jboss.org/byteman - the download includes a programmer's guide. There's also a user forum for asking questions here: http://www.jboss.org/index.html?module=bb&op=viewforum&f=310, and a jboss.org JIRA project for submitted issues and feature requests here: https://jira.jboss.org/jira/browse/BYTEMAN

A Simple Example

The remainder of this post describes a simple example, on the scale of the classic "hello world" example, of using Byteman to insert an exception into a running application.

Let's start by defining the exception that we will inject into our application:

1  package sample.byteman.test;
2
3  /**
4   * Simple exception class to demonstrate fault injection with byteman
5   */
6
7  public class ApplicationException extends Exception {
8
9    private static final long serialVersionUID = 1L;
10   private int intError;
11   private String theMessage = "hello exception - default string";
12
13   public ApplicationException(int intErrNo, String exString) {
14       intError = intErrNo;
15       theMessage = exString;
16   }
17
18   public String toString() {
19       return "**********ApplicationException[" + intError + " " + theMessage + "]**********";
20   }
21
22  } /* class */

There's nothing complicated here, but note the string that is passed to the exception constructor at line 13.

Now, let's define our application class:


1  package sample.byteman.test;
2
3  /**
4   * Simple class to demonstrate fault injection with byteman
5   */
6
7  public class ExceptionTest {
8
9      public void doSomething(int counter) throws ApplicationException {
10        System.out.println("called doSomething(" + counter + ")");
11            if (counter > 10) {
12               throw new ApplicationException(counter, "bye!");
13            }
14            System.out.println("Exiting method normally...");
15    } /* doSomething() */
16
17    public static void main(String[] args) {
18        ExceptionTest theTest = new ExceptionTest();
19        try {
20            for (int i = 0; i < 12; i ++) {
21                theTest.doSomething (i);
22            }
23        } catch (ApplicationException e) {
24            System.out.println("caught ApplicationException: " + e);
25        }
26    }
27
28 } /* class*/

The application instantiates an instance of ExceptionTest at line 18, then runs the doSomething method in a loop until a counter is greater then 10. Then it raises the exception that we defined earlier. When we run the application, we see this output:

java -classpath bytemanTest.jar sample.byteman.test.ExceptionTest
called doSomething(0)
Exiting method normally...
called doSomething(1)
Exiting method normally...
called doSomething(2)
Exiting method normally...
called doSomething(3)
Exiting method normally...
called doSomething(4)
Exiting method normally...
called doSomething(5)
Exiting method normally...
called doSomething(6)
Exiting method normally...
called doSomething(7)
Exiting method normally...
called doSomething(8)
Exiting method normally...
called doSomething(9)
Exiting method normally...
called doSomething(10)
Exiting method normally...
called doSomething(11)
caught ApplicationException: **********ApplicationException[11 bye!]**********

OK. Nothing too exciting so far. Let's make things more interesting by scripting a Byteman rule to inject an exception before the doSomething method has a chance to print any output. Our Byteman script looks like this:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example - throw an exception
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND buffer = 0
9   IF TRUE
10     DO throw sample.byteman.test.ApplicationException(1,"ha! byteman was here!")
11  ENDRULE

Line 4 - RULE defines the start of the RULE. The following text on this line is not executed
Line 5 - Reference to the class of the application to receive the injection
Line 6 - And the method in that class. Note that since if we had written this line as "METHOD doSomething", the rule would have matched any signature of the soSomething method
Line 7 - Our rule will fire when the PrintStream.println method is invoked
Line 8 - BIND determince values for variables which can be referenced in the rule body - in our example, the recipient of the doSomething method call that triggered the rule, is identified by the parameter reference $0
Line 9 - A rule has to include an IF clause - in our example, it's always true
Line 10 - When the rule is triggered, we throw an exception - note that we supply a string to the exception constructor

Now, before we try to run this run, we should check the its syntax. To do this, we build our application into a .jar (bytemanTest.jar in our case) and use bytemancheck.sh

sh bytemancheck.sh -cp bytemanTest.jar byteman.txt
checking rules in sample_byteman.txt
TestScript: parsed rule Simple byteman example - throw an exception
RULE Simple byteman example - throw an exception
CLASS sample.byteman.test.ExceptionTest
METHOD doSomething(int)
AT INVOKE PrintStream.println
BIND buffer : int = 0
IF   TRUE
DO   throw (1"ha! byteman was here!")

TestScript: checking rule Simple byteman example - throw an exception
TestScript: type checked rule Simple byteman example - throw an exception

TestScript: no errors

Once we get a clean result, we can run the application with Byteman. To do this, we run the application and specify an extra argument to the java command. Note that Byteman requires JDK 1.6 or newer.

java -javaagent:/opt/Byteman_1_1_0/build/lib/byteman.jar=script:sample_byteman.txt -classpath bytemanTest.jar sample.byteman.test.ExceptionTest

And the result is:

caught ApplicationException: **********ApplicationException[1 ha! byteman was here!]**********

Now that the Script Works, Let's Improve it!

Let's take a closer look and how we BIND to a method parameter. If we change the script to read as follows:


1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example - throw an exception
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND counter = $1
9   IF TRUE
10   DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!")
11   ENDRULE

In line 8, the BIND clause now refers to the int method parameter by index using the syntax $1. This change makes the value available inside the rule body by enabling us to use the name "counter." The value of counter is then supplied as the argument to the constructor for the ApplicationException class. This new version of the rule demonstrates shows how we can use local state as derived from the trigger method
to construct our exception object.

But wait there's more! Let's use the "counter" value as a counter.

It's useful to be able to force an exception the first time a method is called. But, it's even more useful to be able to force an exception at a selected invocation of a method. Let's add a test for that counter value to the script:


1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example 2 - throw an exception at 3rd call
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND counter = $1
9   IF counter == 3
10   DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!")
11   ENDRULE

In line 9, we've changed the IF clause to make use of the counter value. When we run the test with this script, the first 2 calls to doSomething succeed, but the third one fails.

One Last Thing - Changing the Script for a Running Process

So far, so good. We've been able to inject a fault/exception into our running application, and even specify which iteration of a loop in which it happens. Suppose, however, we want to change a value in a byteman script, while the application is running? No problem! Here's how.

First, we need to alter our application so that it can run for a long enough time for us to alter the byteman script. Here's a modified version of the doSomething method that waits for user input:

1    public void doSomething(int counter) throws ApplicationException {
2       
3    BufferedReader lineOfText = new BufferedReader(new InputStreamReader(System.in));
4        try {
5            System.out.println("Press <return>");
6            String textLine = lineOfText.readLine();
7        } catch (IOException e) {
8            e.printStackTrace();
9        }
10       
11       System.out.println("called doSomething(" + counter + ")");
12       if (counter > 10) {
13           throw new ApplicationException(counter, "bye!");
14       }
15       System.out.println("Exiting method normally...");
16   }

If we run this version of the application, we'll see output like this:


Press <return>

called doSomething(0)
Exiting method normally...
Press <return>

called doSomething(1)
Exiting method normally...
Press <return>

called doSomething(2)
Exiting method normally...
caught ApplicationException: **********ApplicationException[3 ha! byteman was here!]**********

Let's run the application again, but this time, don't press <return>. While the application is waiting for input, create a copy of the byteman script. In this copy, change the IF clause to have a loop counter set to a different value, say '5.' Then, open up a second command shell window and enter this command:

Byteman_1_1_0/bin/submit.sh sample_byteman_changed.txt

Then, return to the first command shell window and start pressing return, and you'll see this output:


Press <return>
redefining rule Simple byteman example - throw an exception

called doSomething(0)
Exiting method normally...
Press <return>

called doSomething(1)
Exiting method normally...
Press <return>

called doSomething(2)
Exiting method normally...
Press <return>

called doSomething(3)
Exiting method normally...
Press <return>

called doSomething(4)
Exiting method normally...
caught ApplicationException: **********ApplicationException[5 ha! byteman was here!]**********

So, we were able to alter the value in the original byteman script, without stopping the application under test!

Pitfalls Along the Way

Some of the newbee mistakes that I made along the way were:

Each RULE needs an IF clause - even if you want the rule to always fire
The methods referenced in a RULE cannot be static - if they are static, then there is no $0 (aka this) to reference
Yes, I had several errors and some typos the first few times I tried this. A syntax checker is always my best friend. ;-)

Closing Thoughts

With this simple example, we're able to inject injections into a running application in an easily automated/scripted manner. But, We've only scratched the surface with Byteman. In subsequent posts, I'm hoping to explore using Byteman to cause more widespread havoc in software testing.

References

[1] http://en.wikipedia.org/wiki/Fault_injection

[2] http://www.jboss.org/byteman

(Special thanks to Andrew Dinn for his help! ;-)

Christmas? Already????

2009-10-01T18:43:00.000-07:00

The stores are already setting up for Christmas. This is very depressing as it is only October.

But - it's never too early for some excellent Christmas music. Check this out:

Christmas Music by Magnatune Compilation

New Post - in a new Blog!

2009-10-01T16:44:00.000-07:00

I just posted to a new blog - the JBoss ESB project blog:

http://jbossesb.blogspot.com/2009/10/content-based-routing-in-jbossesb-just.html

And - that post is on DZone, too!

2009-09-16T18:31:00.001-07:00

Here it is:

http://soa.dzone.com/articles/works-great-right-out-box

New SOA Platform Blog Post

2009-09-15T06:38:00.000-07:00

I just finished this new post to the SOA Platform blog:

http://jboss-soa-p.blogspot.com/2009/09/works-great-right-out-of-box.html

It's a great thing when a product works "right out of the box!" ;-)

Running Eclipse Plugin Tests with SWTBot in Headless Mode

2009-07-23T18:23:00.000-07:00

This is a followup to my previous post on SWTBot. Headless mode is very useful in running tests outsode of eclipse through Ant or from the command line.

Headless?

Well, it's not exactly headless. A better title would be "running Eclipse Plugin Tests with SWTBot with Ant or From The Command Line." The tests are not run from an Eclipse workbench, but they require Eclipse, and actually open up a workbench when they are run.

Install

Here's the place to start: http://wiki.eclipse.org/SWTBot/Ant

Download the "Headless Testing Framework" from here: http://www.eclipse.org/swtbot/downloads.php and install it into your eclipse /plugins dir. You need to install both the junit4.headless and optional.junit4 files. For example:

org.eclipse.swtbot.eclipse.junit4.headless_2.0.0.371-dev-e34
org.eclipse.swtbot.ant.optional.junit4_2.0.0.371-dev-e34.jar

Important note: Be sure to install only these files for either JUnit3 or JUnit4. If you install the files for both versions of JUnit, you'll see many class cast exceptions.

Configuration

Be sure to define a class to serve as a test suite for your tests. For example;

package org.company.test;

import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
 
@RunWith(Suite.class)
@SuiteClasses( { TestPositive.class, TestNegative.class })
public class AllTests {
}

Setup

Export your plugin tests (as a plugin) and install it into the /plugins directory of your eclipse installation.

Running with Ant

Build a build.xml file that looks like this:


<?xml version="1.0" encoding="UTF-8" ?>

<project name="testsuite" default="run" basedir=".">
    <property name="eclipse-home" value="/opt/local/eclipse_swtbot/eclipse" />
    <property name="plugin-name" value="org.company.test" />
    <property name="test-classname" value="org.company.test.AllTests" />
    <property name="library-file" value="${eclipse-home}/plugins/org.eclipse.swtbot.eclipse.junit4.headless_2.0.0.371-dev-e34/library.xml" />

    <target name="suite">

        <property name="jvmOption" value=""></property>
        <property name="temp-workspace" value="workspaceJuly23" />
        <delete dir="${temp-workspace}" quiet="true" />

        <ant target="swtbot-test" antfile="${library-file}" dir="${eclipse-home}">
            <property name="data-dir" value="${temp-workspace}" />
            <property name="plugin-name" value="${plugin-name}" />
            <property name="os" value="linux" />
            <property name="ws" value="gtk" />
            <property name="arch" value="x86" />
            <property name="classname" value="${test-classname}" />
            <property name="vmargs" value="-Xms128M -XX:MaxPermSize=512m -Xmx512M" />
        </ant>
    </target>

    <target name="cleanup" />

    <target name="run" depends="suite,cleanup">
    </target>

</project>

Important note: The supported values for os, ws (workspace) and arch are:

os: win32/linux/macosx
ws: win32/wpf/gtk/cocoa/carbon
arch: x86/x86_64

Important note: Be sure to specify a non-existent workspace name for temp-workspace as this will be overwritten when the test is run.

Running from the CLI

export ECLIPSE_HOME=/opt/local/eclipse_swtbot/eclipse
export TEST_CLASS=org.company.test.AllTests

java -Xms128M -Xmx368M -XX:MaxPermSize=256M -DPLUGIN_PATH= -classpath $ECLIPSE_HOME/plugins/org.eclipse.equinox.launcher_1.0.101.R34x_v20081125.jar org.eclipse.core.launcher.Main -application org.eclipse.swtbot.eclipse.junit4.headless.swtbottestapplication -data workspace formatter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter,$ECLIPSE_HOME/$TEST_CLASS.xml formatter=org.apache.tools.ant.taskdefs.optional.junit.PlainJUnitResultFormatter -testPluginName org.jboss.qa.guvnor.egt -className $TEST_CLASS -os linux -ws gtk -arch x86 -consoleLog -debug

Test Output

It can be hard to find in all the output, but it's there:

Testcase: checkPerspective took 3.529 sec
Testcase: canConnectToRepo took 5.784 sec
Testcase: goIntoBackHomePropertiesTest took 34.526 sec
Testcase: doubleClickTest took 51.74 sec
Testcase: cannotConnectBadPath took 6.701 sec
Testcase: cannotConnectBadPassword took 7.747 sec

Other Useful Links

Ketan added some movies to the SWTBot site in July 2009. Here's one on running SWTBot from Ant: http://download.eclipse.org/technology/swtbot/docs/videos/beginners/SWTBotHeadlessTestingForNovices

Another DZone post!

2009-07-23T08:54:00.001-07:00

The recent post on Gateways and Notifiers in the SOA Platform was just reposted to DZone here:

http://soa.dzone.com/news/soa-platform-gateways-and

A new post for the SOA Platform blog

2009-07-20T13:10:00.000-07:00

Just added a new post here: http://jboss-soa-p.blogspot.com/2009/07/soa-platform-gateways-and-notifiers.html

Listeners and notifiers are very useful things to integrate apps together through the platform. They are actually easy to use, and are very powerful too!

My first post to DZone!

2009-07-13T09:12:00.000-07:00

And - the nice folks at DZone picked up the Content Based Routing post too:

http://soa.dzone.com/news/when-content-knows-way-content?mz=3006-jboss

Another post to the Red Hat / JBoss SOA Platform Blog

2009-07-09T06:38:00.000-07:00

I just finished my 2nd post to the SOA Platform blog here:

http://jboss-soa-p.blogspot.com/2009/07/when-content-knows-way-content-based.html

The diagram towards the end would have really helped me understand content based routing when I first heard of it. Maybe this will help some other newbies along the way... ;-)

Filling in the Hole in the UI Automation Tool Set

2009-06-19T19:21:00.000-07:00

I've never been much of a UI test automation person as I've generally lived on the server-side of things. But, the past couple of years, I've found great set of open-source UI testing tools. In the past few days, I've found a tool that fills in what had been a gap in test tool coverage - automating Eclipse plugins.

The set of tools consists of:

For GNOME Applications - Dogtail

Dogtail developed by Zack Cerza and is written in Python (as are the tests that you write with it) and can be used to automate GNOME-based applications. The design of Dogtail is very interesting as uses accessibility technologies to communicate with desktop applications. It uses accessibility-related metadata to create an in-memory model of the application's GUI elements.

Dogtail can be found here: https://fedorahosted.org/dogtail

(There's also an entry in Wikipedia here: http://en.wikipedia.org/wiki/Dogtail that has links to the user documentation that I wrote for Dogtail. This was published as a series of articles in Red Hat Magazine http://magazine.redhat.com)

For Web Applications - Selenium

With Selenium, you can record or write test scripts that manipulate the web application to be tested through the browser. The tests can be written in HTML or Java or Python or other languages. There's also a record/playback mechanism.

Selenium can be found here: http://seleniumhq.org

For Eclipse plugins - SWTBot

I recently was able to fill in a long-empty hole in this set of tests when a co-worker of mine pointed me at a new tool named SWTBot. SWTBot is developed by Ketan Padegaonkar and automates Eclipse plugin testing.

SWTBot can be found here: http://www.eclipse.org/swtbot

SWTBot makes it very easy to build Java tests for Eclipse plugins. Here's a test code fragment that creates a new project:

SWTWorkbenchBot bot = new SWTWorkbenchBot();
bot.menu("File").menu("New").menu("Project...").click();
bot.tree().select("Java Project");
bot.button("Next >").click();
bot.textWithLabel("Project name:").setText("testProject");
bot.button("Finish").click();

SWTBot is very new; it's in the incubation phase of its development as an Eclipse project. The only problem that I've had with SWTBot so far is that it took me a little while to locate a set of example programs. I did find a good set here: http://code.google.com/p/swtbot-examples

So - to sum things up, with these tools, there's test coverage for desktop, web, and Eclipse-based apps. And that makes for a great tool-set!

Contributing to the Red Hat / JBoss SOA Platform Blog!

2009-06-15T08:54:00.000-07:00

Just added a new post to the JBoss SOA Platform blog here:

http://jboss-soa-p.blogspot.com/2009/06/hanging-together-on-soa-platform.html

There's a feed for it here: http://www.jboss.org/feeds/view/soap?from=0

'Hoping to contribute to this blog on a regular basis going forward...

(No - you're not going blind - the same post - the subject is integrations - was in this blog until a few minutes ago. ;-)

grep - In living Color

2009-05-12T19:48:00.000-07:00

In my last blog post, I mentioned that in performing software testing you tend to look in log files for clues as to the root cause of bugs. grep is a great tool for doing this, but it does have one drawback. If the lines of text in a log file that you are grep'ing through are very long, you can end up with output that looks like this:

grep here-is-the-bad.jar server.log

22:05:01,485 DEBUG [ServerInfo] class.path: /opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes:/usr/lib/here-is-the-bad.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes

Oh. Of course. There's the problem. What? You can't see it? ;-)

Let's try that grep command again. But this time, let's use grep's color option*. This will highlight what we're looking for:

grep --color=auto here-is-the-bad.jar server.log

22:05:01,485 DEBUG [ServerInfo] class.path: /opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes:/usr/lib/

here-is-the-bad.jar

:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes

Now, that's more like it! ;-)

* Special thanks to Ralph for pointing me at grep's color option!

Watching Things...with Linux's "watch"

2009-05-08T20:00:00.000-07:00

When you test software, you tend to spend a lot of your waking hours (and some sleeping hours too) debugging problems. This involves looking around the system undertest for clues as to the root causes of failures.

One trick that I constantly use is to execute the "tail -f" command on system or server or process log files while I'm running tests. The tail command displays the last (10) lines of the file. Adding the -f option results in the file contents to be displayed as they are written to the log file. The result is that I can watch the log file as a real-time window into what's happening on the system under test.

I recently learned[1] about another useful tool; the "watch"[1] command. What watch does is to execute the shell command that you specify at the interval that you specify. What makes watch useful, though. is that is displays its output full-screen, and refreshes the full screen display, so you can don't have to scroll the display for the latest results.

For example, if you wanted to keep an eye on system memory use, you could execute this command to have watch display memory usage information and update the display every 5 seconds:

watch -n 5 cat /proc/meminfo

And see something like this:


Every 5.0s: cat /proc/meminfo                           Fri May  8 22:43:53 2009

MemTotal:      2049932 kB
MemFree:        948244 kB
Buffers:         77540 kB
Cached:         535776 kB
SwapCached:          0 kB
Active:         460124 kB
Inactive:       476236 kB
HighTotal:     1153728 kB
HighFree:       314396 kB
LowTotal:       896204 kB
LowFree:        633848 kB
SwapTotal:     2621432 kB
SwapFree:      2621432 kB
Dirty:              88 kB
Writeback:           0 kB
AnonPages:      323000 kB
Mapped:          67700 kB
Slab:            53548 kB
PageTables:       5064 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3646396 kB
Committed_AS:   904128 kB
VmallocTotal:   114680 kB
VmallocUsed:      7748 kB
VmallocChunk:   106612 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     4096 kB

Or, if you wanted to only watch the amount of free memory, you could execute this variation on that same command:

watch -n 5 'cat /proc/meminfo | grep MemFree'

And see something like this:


Every 5.0s: cat /proc/meminfo | grep MemFree            Fri May  8 22:47:10 2009

MemFree:        932252 kB

As I said, when you're debugging a test, or a potential bug uncovered by a test, you have to watch multiple source of information on the system you're testing. The watch command can be a very useful tool to help you see specific pieces of information in the flood of information that you may have to search through.

References:

[1] http://linux.about.com/library/cmd/blcmdl1_watch.htm

* Special thanks to Ralph for pointing me at watch!

Life as a Scientist

2009-03-30T11:50:00.000-07:00

A great tool to analyze your blog: http://typealyzer.com/

(Found this tool referenced at Mark Little's blog: http://markclittle.blogspot.com/)

Stress Testing: Don't spend your life waiting for a moment that just don't come, Well, don't waste your time waiting...

2009-03-21T17:11:00.000-07:00

I remember the situation distinctly, even though it happened over a decade ago. The product in question was a 2nd generation server. The 1st generation was successful, but like many 1.0 projects, parts of it were, well, let's just say "immature." As a result, the product was buggy and difficult (and expensive!) to maintain. The goals of the 2nd generation product were ambitious; to correct the product's design flaws, extend its feature set, and improve its overall performance and reliability.

After a long development cycle, testing started and we were dismayed to find that the re-architecting effort had completely destabilized the product. We immediately started finding memory leaks, random crashes, loss of data, and basic functions that we just plain broken. The early test results caused a large amount of angst, and also caused us to adopt a very defensive posture. We reorganized testing into discrete stages, and we determined to not advance from one stage to the next until we had established a consistent baseline of reliable product operation.

As the early stages of functional testing progressed, product instability continued, and large numbers of bugs were found, resolved, and verified. Eventually, the product's stability improved, but there serious bugs were still periodically found, so we held to our defensive posture and did not progress from functional to system-wide or stress/performance tests until functional testing was complete. When we finally determined that the product was stable enough for stress testing, the project schedule had been extended multiple times. When stress testing started, began finding still more bugs related to product's inability to function under a load over time or to handle high levels of traffic. Once again, the project schedule had to be extended as large numbers of bugs were found, resolved, and verified.

What went wrong on this project? Well, quite a bit. The planning was overly optimistic, the design was not properly reviewed or understood, unit testing was inadequate, at one point, we even discovered that if every port on the server was physically cabled and the system was not properly rack-mounted, it would tip over and fall on the floor. In addition, by being too conservative in our test planning, and too rigid in our first reaction to the system's initial poor performance, we may have delayed the discovery of some serious problems.

What should we have done differently? We should have started some stress testing as early as could find situations and product configurations that would support it. The first stress tests could have taken the form of low levels of traffic running over an extended period of time, and not necessarily a massive system load at the start. By finding tests that the product could support, we could have established a baseline of performance to measure against as the product's many bugs were resolved (and its code was changed). By waiting for "perfect" conditions that actually arrived very late in the product's development, we delayed tests that could have been run under "good" conditions.

There's a saying that, "if you see blue sky, go for it."[1] It can be like that in software testing too. If you wait for perfect conditions to run some tests, you may end up waiting for a long time, and that waiting may result in bugs being found at a late and probably inconvenient time. When should you try stress testing a product? As early as you can. Performance related problems can be difficult and time consuming to resolve. The sooner you find them, the better. So, if you see some blue sky among the bugs early in a project test cycle, go for it...

(Special thanks to Pavel!...and Mr. Springsteen...)

References:

[1] http://www.pbs.org/wgbh/amex/kennedys/filmmore/pt.html

Parameterized Testing? But wait, there's more! (Inversion of Control/Dependency Injection, that is)

2009-02-16T06:27:00.000-08:00

When you spend most of your waking hours (and, truth be told, some of you sleeping hours too) involved in software engineering, you can find yourself speaking a language foreign to friends and relatives. In my own case, most of my friends and family have no experience in or with software design, development, or testing. They tend to see my software engineering experience as useful, but also something out of the ordinary.

This past Monday, for example, a good friend of mine, a lawyer by profession, telephoned me late at night, desparate to rescue his home computer from the malware that he had inadvertently downloaded. I managed to walk him through the steps necessary to get his computer working again. That is, I thought I had, until a couple of nights later, he called me again asking my advice in buying a new computer.

A few days later, I was reading a printed copy of Martin Fowler's paper "Inversion of Control Containers and the Dependency Injection pattern" [1] when a friend of mine commented, "inversion of control?" Dude, that's the story of my life. My kids rule the house."

The term "dependency injection" can at first be bit more intimidating than the reality of what it is and how it's used. The simplest explanation that I've ever seen for it is in James Shore's excellent blog "The Art of Agile" [2] when he states that:

'...dependency injection means giving an object its instance variables..'

OK. This is all interesting, but what does it have to do with software testing? Quite a bit, actually. In the blog entry from a few weeks ago titled "Parameterized Testing - Very Easy with TestNG's DataProvider" we talked about designing tests that accept parameters, so that the same test could be run against multiple sets of data. In this post, we'll revisit that test design pattern.

But first, let's take a deeper look at inversion of control and dependency injection.

To begin, let's disect the phrase "inversion of control." In this context, who wants control, and just how is this control inverted?

In an object oriented language such as Java, you solve programming problems by designing classes. The classes contain variables (to hold data) and methods (functions that manipulate the data). To actually perform the tasks that you want the programs to do, you create discrete instances of the objects. When you create an object instance, you work with instances of the variables.

Let's say that you've defined a Java class of type Car, and within the class you set a Manufacturer variable to, say, "Audi."

What have we just done in terms of control and dependencies?

* Who has control? The "Car" class has the control. In order to build a new car, it defines a variable of type "Manufacturer." Or more precisely it defines an instance variable of type "Manufacturer." In this case, the class only allows for a Manufacturer of "Audi."

* The "Car" class needs the "Manufacturer" variable to do its work. In other words, it depends on this variable. That's right, this instance variable is a "dependency."

However, there's a problem with the "Car" class. How can build a test for the class, and test multiple car manufacturers? To build a test for Toyota, for example, we will have to build another class that's almost identical to the "Car" class, except that it will create a different "Manufacturer" object.

The answer is easy, of course, we just add another constructor to the Car class:


  public Car(Manufacturer targetManufacturer) {
    carManufacturer = targetManufacturer;
  }

So, now what have we just done in terms of control and dependencies?

* The value of the dependency, that is, the "carManufacturer" variable is not hardwired into the "Car" class. It is "injected" into the "Car" class through its new constructor.

* Who has control now? Not the "Car" class anymore. The class can now be used to create any type of car. Who controls the decision of the type of car to create? The program (or test program) that creates the instance of the "Car" class. That's the "inversion" of control.

One important use of this model is that it emables us to isolate the class being tested. For example, if we want to test the Car class with a mock or stub[3] version of the Manufacturer class, we can pass that stub or mock object to the Car class during the test. This will let the test concentrate on verifying the operation of the Car class, independent of the function (or lack of function) provided by the Manufacturer class.

When you couple this design pattern with ability to pass a parameter into a test, such as is supported in TestNG, this enables us to inject the dependecies into the classes being tested. This makes it possible for the tests to control not only the test data, but also the defined profile and characteristics of the classes under test. And to do it without hardcoding data into the tests or the classes under test. Now, that's control!

References:

[1] http://martinfowler.com/articles/injection.html

[2] http://jamesshore.com/Blog/Dependency-Injection-Demystified.html

[3] Next Generation Java Testing by Cedric Beust and Hani Suleiman, pages 95-96. http://testng.org/doc/book.html

And not a drop to drink - Leak and Soak Tests

2009-02-07T18:27:00.000-08:00

Owning an old house has some advantages. You can enjoy its period architectural details and perhaps find some hidden treasures. When we did some house renovations, we found newspapers from 1923 in one wall of the house. It was quite a surprise to be able to read contemporary sports coverage of Babe Ruth[1]! Owning an old house also means that you are always performing maintenance. I remember late one cold January night when I woke up to the sound of water dripping. It wasn't the bathroom, it was a vintage cast-iron radiator drippig water through the ceiling. Nothing had changed in the water pressure, but the washers in the radiator had just been worn down by prolonged use. Another time, a massive rainstorm resulted in a leak in the roof. The roof had performed well for years of less severe rain storms or equally severe, but shorter downpours, but the extended high level of rain simply soaked through it.

In the first case, the problem wasn't that the level of water traffic had increased beyond what the radiator "system" could manage, it was the accumulated damage of a relatively low level of traffic over time. In the second case, the problem was that the roof "system" was overwhelmed by prolonged exposure to a high level of traffic.

OK - what do all these expensive home repairs have to do with software testing? Just this; when you go about designing stress tests for your product or application, you should include "leak" and "soak" tests* in your plan.

When some people approach stress test planning, they treat the tests as a blunt instrument with which they try to assault the system under test. This type of scorched earth approach to testing can result in system failures being encountered, but these failures can be difficult to diagnose and reproduce. A combination of disciplined leak and soak tests can help you better identify the root causes of stress related system failures.

How do leak and soak tests differ? The differences can be thought of in the terms of the radiator and roof failures that I mentioned above.

In a leak test, you tend to run the system under a manageable, and tracable level of traffic for an extended period, to determine of the accumulated weight of usage causes a system failure. The classic case is when you are looking for a memory leak. The taffic load should not be extreme.

It may be that each system action results in a very small memory leak, or in the premanent allocation of some other system resource. If you run this test once, the leak may occur, but in the context of a system with several GB of memory, and a process that is using several MB of memory, you might not notice that the memory or system resource is not freed up when the test completes. But, if you repeat the test, and observe the process under test a tool such as JBoss Profiler[2], and observe the system with utilities such as top, sar, or vmstat, then you may be able to spot a trend of when system resources are used and not released. A great way to begin a leak testing regimen, is by observing the memory and system resource use of the software under test in an idle state. Just start up the server or application product under test and then leave it alone for an extended period. You may find that it's use of system resources increases over time, just through its self-maintenance, even when it is not actively processing user or client requests.

So, in a leak test, the key variable in the test equation is time.

In contrast, a soak test, hits the system under test with a significant test load, and maintains that load over an extended period of time. A leak test may involve running traffic through the system under test on only one thread, but a soak test may involve stressing the system to its maximum number of concurrent threads. This is where you may start to see inter-thread contention issues or problems with database connection pools being exhausted. If you can establish a reliable baseline of system operation with a leak test that runs over an extended period of time, then you can move onto more aggressive tests such as soak tests. If, however, a leak test exposes system failures, then you would probably encounter the same types of failures with soak tests. But, the level of traffic used in a soak might make these failures harder to diagnose.

So, in a soak test, the key variables in the test equation are both time and a sustained high load level.

To sum it up, if a leak test is passive aggressive in nature, a soak test is just plain aggressive.

* Dzięki Jarek!

References:

[1] http://en.wikipedia.org/wiki/Curse_of_the_Bambino
[2] JBoss Profiler - http://www.jboss.org/community/docs/DOC-10728

Parameterized Testing - Very Easy with TestNG's DataProvider

2009-01-31T18:52:00.000-08:00

I was starting work on some parameterized JUnit tests the other day, when a co-worker*, someone who has a uncanny knack of reducing my complicated questions to simple answers, suggested, "Why don't you try a TestNG DataProvider instead?"

I was already trying to write the tests in Groovy, a scripting language with which I had very little experience, so the prospect of also trying a new test framework was not exactly appealing. But, I decided to give it a try. After a couple of hours of trial and error and self-inflicted syntax errors, I came to the conclusion that a TestNG DataProvider was the way to go. DataProviders are easy to use, and the coding is very straightforward.

TestNG[1] was developed by Cedric Beust and Alexandru Popescu in response to limitations in JUnit3. Actually, "limitations" may be too strong a word. In their book "Next Generation Java Testing"[2], they take pains to say that they developed TestNG in response to "perceived limitations" in JUnit3. In some cases, these were not limitations, but rather, design goals in JUnit that were in conflict to some types of tests. For example, the manner in which each JUnit test case re-instantiates the test class for a "clean" starting point. In this regard, TestNG supports test models beyond unit tests, for example, tests that are inter-dependent.

TestNG provides a couple of ways to support passing parameters to test cases. The parameters can be passed to the test cases through properties defined in testng.xml, or with the DataProvider annotation. DataProviders support complex passing complex objects as parameters. Here's how it works:

You define a method associcated with the "@DataProvider" annotation that returns an array of object arrays to your test cases.

Hang on - an array of arrays? Why is that needed? Here's why - each array of objects is passed to the test cases. In this way, you can pass an array of multiple parameters, each of whatever object type you want to the test cases. It's like this, say you want to pass a String and an Integer[3] to a test case. The DataProvider method returns an object of this type:

Object [][]

So that with these values:

Groucho, 1890
Harpo, 1888
Chico, 1887

The DataProvider provides:

array of objects for call #1 to test method = [Groucho] [1890]
array of objects for call #2 to test method = [Harpo] [1888]
array of objects for call #3 to test method = [Chico] [1887]

Or: this array of arrays of objects = [array for call #1] [array for call #2] [array for call #3]

Simple, right? Once you get past the idea of an array of arrays. Here's the Groovy code.


package misc

import org.testng.annotations.*
import org.testng.TestNG
import org.testng.TestListenerAdapter
import static org.testng.AssertJUnit.*;

public class DataProviderExample {
 
 /* Test that consumes the data from the DataProvider */
 @Test(dataProvider = "theTestData")
 public void printData(String name, Integer dob) {
  println("name: " + name + " dob: " + dob)
 }
 
 /* Method that provides data to test methods that reference it */
 @DataProvider(name = "theTestData")
 public Object[][] createData ( ) {
  [ 
    [ "Groucho", new Integer(1890) ] , 
    [ "Harpo", new Integer(1888) ] , 
    [ "Chico", new Integer(1887) ] 
    ]  as Object[][]
 } 
 
}

When you run this with TestNG, the output looks like:


[Parser] Running:
  /workspace_groovy/BlogCode/temp-testng-customsuite.xml

name: Groucho dob: 1890
name: Harpo dob: 1888
name: Chico dob: 1887
PASSED: printData("Groucho", 1890)
PASSED: printData("Harpo", 1888)
PASSED: printData("Chico", 1887)

===============================================
    misc.DataProviderExample
    Tests run: 3, Failures: 0, Skips: 0
===============================================

I'll return to Groovy and TestNG in some future blog posts as they are very "groovy" test tools. Well, as Groucho would say, "hello, I must be going..."

References:

[1] http://testng.org/doc
[2] http://testng.org/doc/book.html
[3] This is a very slight variation on the sample shown here: http://testng.org/doc/documentation-main.html#parameters-dataproviders

* Děkujeme vám Jirka! ;-)

Defect-Driven Test Design - What if You Start Where You Want to Finish?

2009-01-24T18:46:00.000-08:00

When I start a new software testing project, I always begin by drafting a test plan. I've always viewed test plans as useful tools in pointing your test imagination in the right directions for the project at hand. The very act of committing your thoughts, knowledge, and the results of your test and project investigation to a written plan enforces a level of discipline and organization on the testing. The adage that "plan is also a verb" is very true with regard to test plans.

A written plan is also a great vehicle to use to collect input from other members of your project team such as your Development team counterparts, from whom you can gather information about what new features may be more at risk, the support team, from whom you can collect information about the issues that are affecting your customers, and from the marketing team, from whom you can collect information about business priorities and customer expectations. The task of conducting a review of your plan with these other members of your project team can be a grueling exercise, as they may question both the means and motivations supporting test design or strategy, but their input, along with project design and functional specifications in a crucial element in your creating an effective plan.

But wait - there's another source of information that you should mine. Your bug database.

Think about it for a minute. Where do you find bugs? New code is always suspect as you haven't seen how it behaves under test. But, where else should you look? In the code where you have found bugs in the past.[1] Odds are, this code, is complex, which means it may be hard to change or maintain without introducing new bugs, or it is subject to constant change as the application under test evolves from release to release, or maybe it's design is faulty, so it has to be repeatedly corrected, or maybe it's just plain buggy. By examining your bug database, you can get a good picture which components in your application have a checkered past.

However, you shouldn't stop there. What you want to do is to create a bug taxonomy[2] for your application.

The term "taxonomy" is typically used to describe the classification of living organisms into ordered groups. You should do the same thing with your bug database. Each bug probably contains information about the component or module in which the bug was found. You can use this data to identify the parts of the application that are at risk, based on their bug-related history. You also have have information as to the the types of bugs that you've found in the past. Maybe you'll find a pattern of user access permission problems, where classes of users are able to exercise functions that should only be available to admin users. Or maybe you'll find a pattern of security problems where new features failed to incorporate existing security standards. These are the areas in which you should invest time developing new tests.

OK, so now you have your past history of application specific bugs grouped into a classification that describes the history of your application. What can you do to make the taxonomy an even more effective testing tool? Move beyond the bugs themselves, to classify the root causes of the bugs, and then attack these root causes on your project. You may find that many bugs are caused by problems in your development, design, documentation, and even your test processes. For example, unclear requirements definitions may result in features incorrectly, or in unnecessary or invalid test environments being used. For an extensive bug taxonomy, including multiple types of requirements-related problems, see the taxonomy created by Boris Beizer in his 1990 book "Software Testing Techniques."[3] This is probably the best known bug taxonomy. It includes bug classifications based on implementation, design, integration, requirements and many other areas.

But, don't stop there. What you have in your taxonomy is a representation of the bugs that you have found. What about the bugs that you haven't (yet) found? Take your taxonomy and use it as the starting point for brainstorming about the types of bugs that your application MAY contain. Expand your taxomony to not only describe what has happened, but what might happen. For example, if you have seen database connection related problems in the past, you could test to ensure that database transactions can be rolled back by the application when its database connection fails. Then take it one step further. If you're having problems with database connections failing, what about other server connections? What about the LDAP server that the application relies on for authentication? How does the application respond if that server is unreachable? Does the application crash, or does it generate an informative error message for the user?

How can you then use the taxonomy that you've built? Giri Vijayaraghavan and Cem Kaner identified another use for your bug taxonomy in their 2003 paper "Bug Taxonomies: Use Them to Generate Better Tests"[4]. Their suggestion is to use the big taxonomy as a check against your test plans. For any bug type that you can define, you should have a corresponding test in your test plans. In other words, if you don't have such a test, then you have a hole in your test planning.

To sum it all up, bugs are the goals of software tests and removing bugs from your product is the goal of your software testing. You shouldn't however look at the bugs as a final destination. They are also a persistent resource and as such, they are part of the institutional memory of your project. Building new tests based on these bugs should be part of your testing. Taking the bugs as a starting point, and expanding them into a taxonomy of both actual and potential bugs should be part of your effort to complete the always "unfinished agenda" of your future testing.

References:

[1] Glenford Myers, The Art of Software Testing, p.11.

[2] http://en.wikipedia.org/wiki/Taxonomy

[3] Boris Beizer, Software Testing Techniques, 2nd edition (New York: Van Nostrand Reinhold, 1990). The bug statistics and taxonomy can be copied and used. I found it on-line here: http://opera.cs.uiuc.edu/probe/reference/debug/bugtaxst.doc

[4] http://www.testingeducation.org/a/bugtax.pdf

What's in A Day? It All Depends on What You Mean by "A Day"

2008-09-01T19:46:00.000-07:00

The end of summer is a good time to think about...time.

Maybe it's just the change of seasons, but the end of summer seems to make everyone especially reflective. We all look back fondly on the warm, sunny summer days, and, here in Boston, we look forward less fondly to another long, cold New England winter. And we also think about all the jobs around the house that we had planned to complete during the summer, but somehow never got around to starting.

I was thinking about time this week, and the first time that I had heard the phrase "COB" (i.e., close of business). It was my first job out of college. In the middle of the day, my boss sent me an email requesting (demanding, actually) that I complete a task "by COB." To be honest, I had absolutely no clue what COB meant. I was a recent college graduate, and was trying very hard to learn and memorized what seemed like an endless stream of acronym related to the company's products and technologies. After I was unable to find a definition for COB in any technical documents, I was able to find a co-worker who, when he stopped laughing, was able to explain the term to me.

It's an interesting phrase, "COB." It implies that there is a closing time for the business conducted, or work performed in any given day.

As I'm writing this blog entry, it's late at night in Boston. (The Red Sox just defeated the Chicago White Sox in a night-time baseball game, so, all's right with my world.) My calendar day is closing, but I just noticed that some of my project team co-workers just returned from lunch. It's already tomorrow for them as they are based in Brisbane. The rest of the project's team is based in China, the UK, the Czech Republic, Germany, Sweden, and elsewhere in the US and EU.

Given the geographic diversity of the project team, is there every a COB? I think that the answer is...rarely. The wide variety of locations involved means that there's almost always someone at work on the project.

It's really not that unique these days to work on a team that is not located in the same place. In order for the team to be successful, everyone has to be aware of everyone else's timezone, and make effective use of email and on-line chat tools to communicate. One useful thing to do is to always refer to time (for example, in the scheduling of video or teleconference meetings) in terms of UTC, so as to avoid errors in converting between timezones.

But, beyond any scheduling or communications tools, I think that the most important things to keep in mind involve how you perceive a couple things.

First, the perceptions of "remoteness." It's natural to view the rest of the world relative to your location, your home, your city, your country. For example, on my project's team, I sometimes think about my UK and EU and APAC co-workers as being in remote locations east of me, and my western US co-workers as being in remote locations west of me. [1] The reality, however, is that on this project, everyone is remote. Some offices host more or fewer team members than other, but there really is no "central" office for the project. So, it's important to keep in mind is that it's not "they" who are remote. We all are. No one is "satellite" or 2nd class team member by virtual of their physical location.

Second, the perception of just what a "day" is. Everyone's personal working day is a finite thing. There's a beginning and an end. With a widely dispersed team, however, if you can organize things properly, the calendar days can blend together into a much more flexible and fluid construct. It's sort of like a ship at sea. The ship doesn't turn off its engines at 5:00PM local time. Instead, the crew is divided into teams that work in shifts that are seamlessly put together. It takes effort and organization to ensure that tasks are handed off cleanly between shifts, both on a ship and a software engineering project. You have to be sure that your work can stand alone if a fellow team member is relying on it at a time when you are unavailable. For example, in testing it is extremely important to fully document all bugs that you find with stack traces, server log files, and example code that illustrates the bug. If any of this information is not included in a bug report, then anyone trying to recreate and fix the bug will have to request the information from you, and it may be several hours until you see this request for additional information. The key is to remember that your "day" doesn't exist in a vacuum, but that it is part of an ongoing stream of time and effort, where calendar days can blend together.

I have to wrap up this post now. I have to send some answers to questions to my friends getting back from lunch in Brisbane, and send some questions to my friends who will be sitting down to breakfast in Brno in a couple of hours. It's all in a day...

References:

[1] A note on Bostonians - some of us still see Boston as the "hub of the universe." http://www.boston-online.com/glossary/hub.html

Self-Inflicted Bugs - The Workaround Trap

2008-08-05T18:56:00.000-07:00

(We'll get back to the "what do I do now?" track in the next few posts. But, hey, it's baseball season!)

The year was 1929.

Legendary baseball manager George Stallings, who had led the 1914 Boston Braves from last place the year before to win the World Series, lay on his deathbed.

When a friend asked him what was hurting him, he reportedly replied, "bases on balls."[1]

This story may require some translation. In American baseball, a "base on balls" refers to a batter being awarded first base if the a batter receives four pitches that are outside of the strike zone, and which he is not obliged to attempt to hit. This is also called a "walk" as the batter does not have to run to be able to reach first base. Walks are extremely frustrating to managers as it is a self-inflicted problem. When your pitcher walks a batter, then you've given your opponent an advantage that they did not have to earn by actually hitting the ball.[2]

That's all very interesting, but what does this have to do with software testing?

If you research software bug taxonomies[3], you'll information on types of bugs such as incorrect requirements bugs, data translation bugs, interface bugs, and so forth. Actually, software bug taxonomies are great test planning tools. If you haven't researched these taxonomies before, then you really should. You'll probably quickly find ways to apply these bug types to your own project test planning.

One type of bug that you may not find in a standard bug taxonomy is the self inflicted bug. I was thinking about self inflicted bugs the other day when I had lunch with some former co-workers and we reminisced about a long ago software project. The project had many types of bugs; reliability, installation and configuration, data conversion, usability, build integrity, etc. And, in the course of trying to deal with these bugs, we inadvertently introduced some self inflicted bugs.

Familarity Breeds Politeness, and Sometimes Missed Bugs

What happened was that over time, we had all became so accustomed to the project's many problems, that we knew that in order to be able to make progress in testing some of the project's features that we would have to avoid certain actions that we knew would to fail. This approach can be useful when some testing is blocked as it enables you to work on the tests that are not blocked. It is, however, a "slipperly slope" as in software testing, once you start working around some of a product's bugs, you can get into a mindset where you forget that both those bugs and other bugs you encounter are BUGS that must be fixed.

I once worked with a very diligent and competent software tester who spent days trying to execute a complex text on the product he was testing. He worked through many scenarios, but the program always failed, When he asked me in frustration, "Why doesn't this work?" my answer was "maybe because it's broken?" He had fallen into the workaround trap. He was trying everything he could think of to help a broken piece of software to work, when what it really needed was for some bugs to be fixed.

In a way, becoming too familar with a product's limitations can be a liability in that you may unconsciously avoid performing certain actions because they will push a program under test beyond its limits and cause a failure. This familarity is a little like the way families sometimes operate. It's similar to the situation of when your uncle comes to Christmas dinner and tells the same unfunny jokes year after year. Everyone smiles and laughs, because "we all know that's how your uncle is." In reality, what you'd like to do is to fix the bug tell him to just keep quiet!

This is actually an important reason why you want to have an independent software testing team; it's very difficult to be hostile toward your own work. If you're developing a software product, your goal is to see it succeed. You may build test suites to verify its operation, but, you may unconsciously avoid building in tests that you know will break it. Software development is a creative process. Testing, however, is destructive.

In the case of our project, we worked around many bugs, and were therefore able to find many other bugs. Some of these workarounds took the form of manual corrections in system configurations that were caused by a buggy installation utility. Our options at the time were to either halt all testing to get the installer fixed, or work around the problems so that we could make progress testing the product. In the process of relying on the workarounds however, we lost track of some of them and ended up having to have them fixed very late in the testing cycle.

What could we have done differently? We tended to track the workarounds in emails or meeting minutes. What we needed to do was to tracking them in our bug tracking system as bugs, and highlight them as high priority bugs that needed to be be fixed. But, most of all, we needed to remember that the workarounds that we put into place were never intended to be part of the product, but they were only temporary constructs. Our goal was that we wanted the project to be able to do without the workarounds. Just like Christmas dinner without your uncle's jokes.

References:

[1] http://www.baseballlibrary.com/ballplayers/player.php?name=George_Stallings_1867

[2] http://mlb.mlb.com/mlb/official_info/official_rules/definition_terms_2.jsp

[3] Software Testing Techniques by Boris Beizer

The First Person to add to Your Team?

2008-07-09T19:42:00.000-07:00

While you'll have to work on the people, tools, and processes tracks simultaneously, we'll take them one at a time in our discussions.

Let's think about the people that you need for your team first. A while ago, I wrote an essay for IBM Developerworks [1] that described some of the characteristics (and some of the characters) that you should look for in software QE engineers in order to build an effective team. I'll expand on some of the ideas from this essay in a later post. The specific subject that I want to address in this post however, concerns the first person that you should try to add to you new team.

The first person that you need to look for is a "grinder."

OK. What's a grinder? And, why do you need one first? Shouldn't the first member of the team be a test framework architect or automation coder? Think back to our hypothetical project situation. You have to juggle the creation of the team, its tools and the project's set of QE processes while you also ship a product. The skill set of the first person on your team has to be that of a general practitioner (sort of a software Swiss Army knife) who can both execute and build tests. But beyond his/her technical skills, you also need to find someone with the right attitude.

Once again - what's a grinder?

It's a sports metaphor. It's sometimes applied to baseball pitchers, but it most frequently is used when describing golfers. In golf terminology, "grinder" describes someone who is faced with adverse conditions and is not playing their best game, but is still able to, through effort and determination, "grind" out a victory. A great example of this happened at last month's US Open. [2] Tiger Woods was playing in his first event after knee surgery. As it turns out. he came back from the surgery before his knee was fully healed and as the tournament proceeded, he was playing in more and more pain. For me, the image of what it means to "grind" was watching Woods experimenting with different swings and shot patterns as he searched for a way to swing without collapsing in pain. In spite of the pain and the difficulty of playing the most difficult course of the entire season, he was able to focus on his goal.

You'll need the first person on your team to have this type of attitude. (You'll also need to be something of a grinder too.) If your first team member is a test architect, he or she may become frustrated by the need to meet short-term product delivery goals instead of being able to concentrate on a longer-term design effort. Likewise, if your first team member is a test automation specialist, he/she may become frustrated by the need to "jump in" and perform manual or exploratory tests. [3]

So - start with a general practitioner, but be sure to find one who can "grind it out."

References:

[1] http://www.ibm.com/developerworks/rational/library/content/RationalEdge/sep04/dimaggio

[2] http://www.usopen.com/en_US/index.html

[3] http://swqetesting.blogspot.com/2007/11/in-praise-of-exploratory-testing.html

Three Tracks

2008-07-09T19:40:00.000-07:00

I was wandering around FUDCon[1] in Boston a couple of weeks ago, and ran into someone who said, "I really like your blog, but the entries have been very short recently."

Ouch.

The awkward thing was that he was correct. I had been unable to contribute much to the blog for a while as I had been concentrating my writing time on Red Hat Magazine[2].

Let's get back to our "now what do I do?" thread. In the last post, we talked about dealing with risk including understanding that risks are always present. OK, you know that your new friend and constant companion is Mr. Risk. Now what do you do?

You have to make progress on (3) tracks:

* People
* Tools
* Processes

We'll examine aspects of each of these tracks in the coming posts to this blog.

References:

[1] http://fedoraproject.org/wiki/FUDCon/FUDConF10

[2] http://www.redhatmagazine.com

Open Source ESB - New Magazine Article

2008-05-30T06:37:00.000-07:00

I mentioned Red Hat Magazine a while ago - it's really a great resource for technical open source content. And, speaking of content, I was able to publish this new article a few days ago:

http://www.redhatmagazine.com/2008/05/22/adapters-for-an-esb

The subject this time is the manner in which the JBoss ESB (Enterprise Service Bus) supports adapters so that apps (including legacy apps) can integrate with the ESB. Very cool stuff.

You'll Never Walk Alone

2008-04-27T19:35:00.001-07:00

Before we start looking finding QE team members, building automated tools, and defining lightweight and repeatable processes, there's one point I want to make. You may be feeling a bit overwhelmed by the scale of the work you have to do in your new position. You may also be feeling a little lonely. But, fear not. You are not alone. You have a new constant companion. A companion that will be right beside you every step of the way. Who or what is this companion?

Risk.

That's right. Risk will always be with you. How you handle risk as it applies to the software under test will determine how successful your testing is. So, how do you handle risk? I think that there are (3) things you have to keep in mind:

1) There is no "Out" - A few years ago, a good friend mine called me to say that he was completely debt free. He had paid of his mortgage and his car. He said that he would "never again" be in debt. Then, his car needed major repairs. He bought a new car. Then, he and his wife decided that their kitchen was outdated. They remodeled. Then they had a baby. You probably get the point. Risk will always be there and you will always have to work to minimize its impact on your project. Something, whether new features, new requirements, new or old bugs, will put the project at risk. Speaking of bugs...

2) There are Always Bugs - Have you found all the bugs in that code? No? Well, don't worry. You never will. The simple fact is that there's no such thing as "bug-free" software. Why is this so? First, on the development side of the equation, you have to deal with changing technology, a complex and often flawed application design, the difficulties inherent in integrating new and existing systems, and so on. Human error is also a huge factor. Although modern application development tools can generate code, people must be involved at some point in the development process. And people make mistakes. On the testing side, you will always have to concentrate on finding the bugs that put your project most at risk. Most at risk today that is...

3) "What's at Risk?" The Better Question to Ask is, "What's at Risk NOW?" - No software project will even be static. To be successful, your project will have to respond to your customers' changing requirements and will have to adapt to and incorporate new technologies. There's a great line from JFK talking about political parties that I used in article to refer to software test planning. The line is that these parties aren't locked in amber, but flow like rivers through time. (Theodore Sorenson actually wrote many of his speeches, but JFK always claimed this line as his own.) It's like that for test plans too. You can try to anticipate everything in advance, but in the course of testing, you always learn something new and have to change the plan to incorporate new or changed tests based on the risks that the project is currently facing. These risks will change over the life of a project. Early in a project's development, the major risk may be that new features are simply buggy, or that integrations between major project subsystems are in conflict, or that the build process is so immature that unintended software changes are introduced. Later on in a project's development, the major risk may be that the original designers of a project have been replaced by new engineers who, in the course of fixing bugs, break the existing design.

So - how do you cope with all this?

First, don't expect that your job will ever be risk free. Your job is to manage that risk.

Second, don't expect that your software will every be bug-free. You have to manage your resources and testing to find the bugs that matter most.

Third, don't expect that you can relax because "all the risks" have been mitigated. In the time that it took you to read this blog post, a new risk was probably either exposed, inadvertently coded, or dreamed up down the hall in marketing. So, don't lock your thinking or your test planning in amber. Think of your project as a river. Maybe even the kind that people go white-water rafting it.

"Now What Do I Do?"

2008-04-07T20:22:00.000-07:00

I'm been thinking of taking this blog in a slightly different direction. The motivation for this change came from the question that is the title of this post.

A couple of years ago, a good friend of mine, a product manager, called me up on a Sunday night asking for help. His software startup company had been growing quickly, as had the number of bugs in its product. They needed to implement a formal testing process, put together a QE team, and build automated tests. And fast. Unfortunately, no one at the startup had ever done this before. My friend was brave enough to say, "hey, I know someone who could help." So - he was placed in charge of the task. I wrote up a couple of pages of ideas for him so that he could get started.

I've thought often of his question as other people have also asked me about how to set up a QE team from scratch. One of them was also at a startup, a couple of others were working on new projects at larger companies, while another had just been promoted, with no advance notice, to manage a non-existent QE team. What they all had in common is that they had to build a new team, find or develop tools, and define a testing process, all while they were also trying to meet an aggressive product release schedule. Sort of like building a bus while it roars down a highway.

Anyway, what I'm going to try to do in the next series of posts to this blog is to answer that question. I'm hoping that this series of posts will be a useful roadmap or reference guide. The scale of the posts will probably be longer than most blog posts, but I'm going to keep the combined total length of the posts shorter than a book. After all, if you're in a situation similar to one my friend found himself, you don't have a lot of time to read!

Great Tool for Swimming with Network Sharks

2008-03-21T18:25:00.000-07:00

I was looking to debug a problem involving clients connecting to FTP servers and was in need of a packet sniffer. A colleague of mine pointed me at an open source tool named "WireShark" (http://www.wireshark.org). This is a great tool. It's a bit like tcpdump, but it includes a beautiful GUI and does a great job at filtering packets and at exporting/importing data. It also run son Linux, Windows and Mac OS X.

Here's a screen shot:

From my favorite e-zine...

2008-03-11T11:10:00.000-07:00

That last post about "what is middleware?" received a nice response from many readers - strangely, no one actually commented on the post.

Anyway - a longer version of the topic was just published in Red Hat magazine:

http://www.redhatmagazine.com/2008/03/11/what-is-middleware-in-plain-english-please/

What is Middleware? Huh? No, now try that again in a language that I can understand...

2008-02-14T19:37:00.000-08:00

I listened to a webcast from JBoss World today with a group of people. After hearing several speakers announce new middleware products and initiatives (as JBoss is the leading force in open source middleware), one of them turned to me and asked, "Just what is middleware?" When I started to describe transaction servers and database connection pool sharing, she held up a hand and asked, "No. I want to know what it is in real world terms, and why it's a big deal."

That got me thinking and sent me to google to look for a short definition of middleware. I found a lot of them, but they mostly were either too vague or too dependent on the reader already having some knowledge about middleware. Then, I found this one:

'...middleware: The kind of word that software industry insiders love to spew. Vague enough to mean just about any software program that functions as a link between two other programs, such as a Web server and a database program. Middleware also has a more specific meaning as a program that exists between a "network" and an "application" and carries out such tasks as authentication. But middleware functionality is often incorporated in application or network software, so precise definitions can get all messy. Avoid using at all costs...'[1]

And that really got me thinking about how to describe middleware and why it matters. What I was searching for was a real-world analogy that would make sense to people with varying levels of computer and software experience. And then, it hit me.

Middleware is plumbing.

There are four ways that this is true.

First, it ties together disparate parts of complex systems.

In your house, you have kitchens, heating systems, bathrooms, washing machines, garden faucets, etc. Each plays an important part in making your house livable. You almost never don't have to worry about not having running water because the plumbing is robust and reliable. It just keeps the water moving through the pipes. Middleware keeps information moving through complex web-based applications. One of its primary tasks is to connect systems, applications and databases together in a secure and reliable way. For example, let's say you bought an over-priced sweater at a store web site last night. What happened? You looked through various sweaters' images, selected color and size, entered a charge card number and that was it, right? Well, behind the scenes, middleware made sure that the store's inventory database showed that sweater in stock, connected to the charge card company's database to make sure that your card was not maxed out, and connected to the shipping company database to verify a delivery date. And, it made sure that hundreds or thousands of people could all shop that site at the same time. Also, while it looked to you like you were looking at one web-site, middleware tied together many different computers, each in a different location, all running the store's e-commerce application, into a cluster. Why is this important? To make sure that you can always get to the store on-line, even if some of these computers are down due to maintenance or power failures.

Second, it's mostly invisible.

You don't generally see much of the plumbing in your house. What you see is the water. As a consumer, you don't see middleware. You see the web sites and the information flow that middleware makes possible.

Third, it provides a standard way of doing things.

If you wanted to build your own plumbing from scratch, you could. But, it's much easier to just buy plumbing fixtures. You, as a software developer, could design and build your own application servers, database connection drivers, authentication handlers, messaging systems, etc. But, these would not be easy to build and maintain. It's much easy to make use of middleware components that are built according to established (and especially open!) standards. In middleware, these standards take the form of libraries of functions that your programs call through well-defined application programming interfaces (APIs). You call these functions instead of having to invent your own.

Fourth and finally, it lets you worry about other things.

When you put an addition onto a house, what do you worry about? Bathroom fixtures, kitchen appliances, flooring, colors, and how to pay for it all. It's a very stressful process. The last thing you want to worry about is whether you want 3/4 inch or half inch pipe, copper or PVC connectors, #9 or #17 solder, etc. With middleware taking care of all the invisible functions, you, again as a software developer, or a business owner, can concentrate on building software to solve your business problems and fulfill your customers' needs.

My father once told me, "when you move to a new town, don't look for a doctor - first thing, you find a good plumber so you can sleep nights." It's like that with middleware. It may be mostly invisible, but it keeps things running so a lot of developers and managers and customers can sleep at night.

Ref:

[1] http://www.salon.com/tech/fsp/glossary/index.html

Untimely Assumptions to Look for in Test Design

2008-02-03T19:28:00.000-08:00

Having no time to blog got me thinking about...time.

In my Sept. 2007 Red Hat Magazine article[1] about software about software communities, I referred to the need to deal with different timezones when your team and community are spread across multiple geographic locations. The folks with whom I'm working these days can attest to my being challenged when it comes to converting times between timezones. I always seem to ask them to attend meetings at inconvenient times for them! ;-)

Anyway, I received an email yesterday that got me thinking about time, and about assumptions that sometimes get made in system design about the availability of "quiet time." The email announced a lab was being powered down at 2:00 AM for some necessary maintenance. The time of day selected was a "quiet time" when the lab systems would be idle.

While I was reading the email, I had to think when 2:00 AM local time was in remote offices as we frequently share test servers in remote locations. As it turned out, the lab outage was not a serious problem based on the location and timezones of people would would be accessing the lab systems.

But, wow, that email caused me to have a sudden flashback to a software project on which I worked several years ago. The project was a voicemail call accounting system. Every day at 2:00AM the system ran several accounting tasks to calculate billing for that day. These billing tasks were large and complex and reduced overall system throughput while the tasks were running. The only problem with the system was that while 2:00 AM might be a quiet time for your own location, it might be a busy time for users accessing the system from elsewhere in the world, and those users would see poor performance when they accessed the system.

I don't think that we ever solved this problem, and I have long since left that company, but it illustrates an interesting problem in developing real-world stress tests. If the system under test includes faulty assumptions about traffic patterns, then unless you are able to identify the implications of these assumptions and attack them, then your testing may be flawed. The discrete tasks that the system has to perform may vary at different times of the day, or days of the week or month.

Let's wrap this up: what are some good faulty design assumption to look for in today's globally networked world?

The notion that the requirements that the system under test must fulfill are unchanging over time. Just as traffic patterns will change over a time period, the tasks performed by the system may vary from hour to hour.

Also, the notion that the system under test is the center of that world. These days, the system under test may be in Bonn, but it may have to simultaneously support people in Boston, Bogata, and Brno. And remember, 2:00 AM arrives at a different moment in time for everyone!

[1] http://www.redhatmagazine.com/2007/09/11/a-tale-of-three-communities/

Need to Slow Down the Blogging for a Bit

2008-01-29T18:14:00.000-08:00

I'm going to have to slow down on the blogging for a while as I want to start to write some new articles for Red Hat Magazine. If you haven't seen the magazine yet, it's really worth a look. (And, I'm not just saying that because I've been lucky enough to have published articles in the magazine.) It's a great resource for information On Linux, open source software, and of course Red Hat and JBoss. And - it's updated daily.

http://www.redhatmagazine.com

OK. I spoke too soon. There's one other resource that I really want to highlight.

http://www.opensourcetesting.org

This is a great site for open source testing tools, ideas and discussions. All open source of course! ;-)

The Father of the Bride Question

2008-01-13T19:52:00.000-08:00

(A follow-up post to "Why Are the Fountains Broken?")

What question is asked by all fathers of the bride when a wedding is being planned? "How much will this cost me - can we make it cost less?"

In a recent post to this blog, I talked about the often forgotten cost of maintenance for automated tests. Let's now talk about how much this maintenance may cost, and how to limit that cost. In other words, how much of my project schedule do I have to devote to maintaining my automated tests? Let's walk through an example. Suppose you are building the first automated tests for a new product. For the sake of this example, let's assume that your test plan calls for you to create (50) automated tests. Let's also assume that you are using an existing test framework such as JUnit or TestNG.

The first test may take you up to a day to create as you will be learning the product under test. You will likely spend a good deal of time simply getting a simple test running. After this, you will probably be able to write perhaps 50% of the tests at the rate of two per day, and perhaps the other 50% at the rate of three to four a day. When you have them completed, you'll need some additional time to refactor and generally clean up the tests. Let's allocate three more days for that.

So, in summary:

the first test = 1 day
the next 24 tests = 12 days
the next 25 tests = 7 days
clean up = 3 days

For a total of 23 days = in other words, perhaps a month.

Now, let's think about maintanence. Let's assume that your new product ships quarterly updates which contain bug fixes and minor new features, and annual major releases. All of these new releases and bug fixes require that you write new automated tests, so over time, your test library grows and grows. But, how much of your time and person resources should you plan to spend on maintaining these tests?

To a large degree, the amount of investment in maintaining the tests will depend on how effective the tests are, and on how much the code actually tested by the tests changes. Let's say that 10% of the code in your project changes in each of your quarterly releases. Where will you spend your time and resources in updating your tests?

First, you will have to review all the tests so that you can determine which tests must be updated, and which new tests must be created. Then you have to design and implement the changes. Let's say that you can review ten tests per day, and that the time required to update these tests is on the scale of one half day for each of the 10% of tests.

So, in summary, maintenance on the first set of tests will cost:

review all the tests = 5 days
update the tests = 3 days

For a total of 8 days. Or, about 25% of the time to write the tests in the first place. If this sounds expensive, well it is! What we need to do is to find a way to reduce the time needed to review all the tests, because remember, the number of tests is always growing.

What's the answer? There's no silver bullet to the mundane task of maintenance. But, part of the answer is the mundane task of documenting the tests so that the specific tests that ought to be changed can be easily found.

This documentation should record not just the design of the tests, but the goals of the tests. This documentation becomes the "institutional memory" of the test automation in that it is persistent, and it outlives any one person's involvement with the project or the tests. Another part of the answer is to keep this documentation in a form that can be easily reviewed, that maps directly to the project requirements' definitions, and is easily edited. Javadoc is a great approach for this as it enables you to keep the test documentation in the actual test source files.

A few years ago, I was involved in a situation where I was dealing locating test coverage for a specific feature within a test suite of several thousand tests. Over the years that it took to develop the tests, the knowledge of the precise actions performed by any one test was lost. The only way to review the tests was to review the source code of each test. It was possible for Development and QE engineers to do this, but other project team members who had project knowledge, but not programming skills, were not able to contriute to the review.

So, why are the fountains broken? Maybe no one planned to or was able to maintain them. Why are the tests outdated? Maybe for the same reasons. But, if you plan for maintenance and make it possible for everyone on the project team can easily review the tests in a form that they can understand, then maybe you can direct your finite test automation resources at the tests that need updating. And in the process save some time and money.

To Make Sure the Software is Functional, Don't Forget the Non-Functional Tests

2008-01-05T16:22:00.000-08:00

When I first heard the term "non-functional test, I thought that it was a joke. I was working for a now-defunct company on a now forgotten product. The product's code was anything by fully "functional," so when I was asked about non-functional tests, it seemed more like an appropriately funny and sarcastic comment than a serious question. It may sound odd when you first hear it, but the classification of tests as "functional" or "non-functional" actually is very logical.

What are functional tests? These are the tests that you build and run to exercise (and exorcise the bugs out of) the code that supports the functions and features of the product under test. In other words, these tests tell you if the product fulfills its functional requirements. What are non-functional tests? The best description that I've seen is that these tests verify how well the product fulfills its functional requirements. A good way to think about this difference between functional and non-functional tests is that the functional tests verify the "whats" and the non-functional tests verify the "hows."

The details of functional tests for a product will be specific to that product. For examples, tests for a text editor will be different from tests for a firewall. The types of non-functional tests that apply to a product will also depend on the characteristics of the product, but will likely include types of tests such as these[1]:

* Compatibility testing
* Compliance testing
* Documentation testing
* Endurance testing
* Load testing
* Localization testing and Internationalization testing
* Performance testing
* Resilience testing
* Security testing
* Scalability testing
* Stress testing
* Usability testing
* Volume testing

So, functional tests or non-functional tests? Which types should you build and run? For a thorough test cycle, you really need both. Hmm. What about dysfunctional tests? I'll get back to talking about these in a later post.

[1] http://en.wikipedia.org/wiki/Non-functional_tests

That time of year again!

2007-12-23T19:49:00.000-08:00

Frohe Weinachten! Boun Natale! Joyeux Noel! Merry Christmas!

(I'll blog about software testing after the holidays!)

Here are some scenes from snowy Boston...

When Less May Be More - A Lighter Weight Test Plan

2007-12-15T19:20:00.000-08:00

I was talking to a couple of colleagues about test plans this week; both the format of the plans and the content.

The IEEE standard is well, a "standard." It may, however, be intimidating to people unaccustomed to software testing. And, it lacks an easy to use construct for differentiating between classes of tests. The discussion of test plans came up this week as one of my colleagues is trying to - gently, but effectively - introduce a structured test process into an organization whose members are largely unfamiliar with it. He wanted very much to take a medical approach to this and "first, do no harm." For him, a lightweight adaptation of the the IEEE standard is a better starting point. We came up with this template for a test plan:

Introduction - what are we doing?
Test Strategy - how are we doing it
Test Priorities - what's most important
Scope - What's tested?
Scope - What's beyond the scope of testing?
Test Pass/Fail Criteria - how do we know that it's good or bad
Test Deliverables - docs and programs that we'll build
Test Cases - Functional Tests
* Tests mapped to product features
Test Cases - Non-functional tests - whichever apply[1]
* Compatibility testing
* Compliance testing
* Documentation testing
* Endurance testing
* Load testing
* Localization testing and Internationalization testing
* Performance testing
* Resilience testing
* Security testing
* Scalability testing
* Stress testing
* Usability testing
* Volume testing
Test Environment/Configurations
Responsibilities - who's doing what
Schedule/Milestones - when are they doing it
Risks and Contingencies - what might go wrong and how we'll handle it
Approvals - do we agree?
References - pointers to background docs
Revision history - why did the plan change and how?
Appendices - anything else?

It's worth noting how this template explicitly separates the descriptions of functional and non-functional tests as the differences between these test class definitions may be new concepts to some of the team members. What's that? Are these test class differences also a new concept to you? I'll discuss this subject in the next post to this blog!

Ref:

[1] http://en.wikipedia.org/wiki/Non-functional_tests

P.S. My Fedora friends have picked up on this test plan outline - it's the basis for their test plan template: https://fedoraproject.org/wiki/QA:Test_Plan_Template

Why Are the Fountains Always Broken? (The forgotten cost of test automation: maintenance)

2007-12-10T18:25:00.000-08:00

I'm writing this post from just outside of Boston. By American standards, Boston is an old city. One of the distinguishing features of the city is its collection of public parks. These parks are dotted with fountains which vary in ages and design from classical and historic to modern and avent-garde. These fountains, however, all share one common characteristic.

They always seem to be broken.

Why is this the case? It is certainly at least partially due to the ravages of harsh New England climate (As I'm writing this, the Boston area is having its second snow and ice storm of the young 2007-2008 winter) on outdoor plumbing. There is, however, I think another possible reason.

When we (we human beings, that is) build something, we often forget about the cost of maintaining it after it is built.

In the case of the fountains, the lack of maintenance may be caused by the fact that in any year's budget, other budget responsibilities such as health care, roads and bridges, and public safety are higher priorities. While it may be possible to attact funding for new construction of exciting or ground breaking public places, maintenance is, in contrast, considered a mundane exercise.

The same pattern can be seen in software enginering. For example, in the development and maintenance of automated software tests.

In the course of developing a plan for automating tests, we consider the investment in time and resources to design, build and debug the tests. Once written, however, the tests will lose their value unless they are maintained and kept in synch with the software project that they are intended to test. As the project procedes from release to release, new features are added, and new tests are needed. But, the existing library of tests may either begin to fail due their falling out of synch with the project code. Worse yet, the existing tests may provide a false sense of security as they may continue to run without error, but fail to actively exercise changed, and therefore, at risk, areas of code.

Automated tests are essential to any software test effort. The tests will pay back the investment you had to make to build them many times over during the life of a project. But, don't think that the tests are "done" just because you've completed building the first version of them. You will have to revisit and review and update the tests to keep them in working order. So, when you build new tests, don't forget to plan for maintenance. Part of this planning involves good test design, so that the tests can be modified as needed. Another part of this planning involves remembering that this maintenance will require time and human effort.

Why are the fountains broken? Maybe because no one planned to or was able to maintain them. Are your automated tests running correctly today? Even if it has been several months and project updates since they were first written or last reviewed? It might be a good time to look for leaks...

Why Doesn't This Work? It's Broken. That's Why!

2007-12-01T19:52:00.001-08:00

Here's a rathole that software test engineers often fall into. When you test software, you frequently encounter situations where some bugs block certain sets or types of tests. Until these bugs are resolved, you often resort to "workarounds" to avoid these problems and to enable you to continue to make testing progress. The danger is that in the course of "working around" these problems, you forget that the workarounds each have at their core an actual bug. And, if these bugs are not correctly resolved, you may never catch them. But, if you don't, your customers probably will.

So, in your earnest attempt to make as much testing progress as you can, even with immature and buggy software, don't forget to remove any workarounds that you put into place. Each workaround should always be treated as a "canary in a coal mine" and as a pointer to a potential serious bug. A good way to deal with these workarounds is to track them in your bug tracking system along with the bugs. You should close out one of these workarounds when the underlying bug is resolved and you can begin to take the road that you wanted to take in the first place! (And that will really make all the difference. My apologies to Robert Frost. ;-)

The Opposite of Fundamentals, the "Ratholes"

2007-12-01T19:49:00.000-08:00

There's a great line from Robert Frost's poem "The Road Not Taken"[1] that I've always thought applied to software engineering and software testing. The line from the poem is: "Two roads diverged in a wood, and I took the one less traveled by, And that has made all the difference."

Sometimes however, the road not taken is not taken for a very good reason! It can be the case that this road can lead down a rathole and is based on bad software design, poor implementation, or tactical mistakes.

As a similar construct to the fundamentals (the "rules") of software testing that I first mentioned in an earlier post, I'm also going to start a series of periodic posts on mistakes that people often make. In other words, the ratholes into which we all fall when we choose the wrong road.

Ref:

[1] http://www.bartleby.com/119/1.html

In Praise of Exploratory Testing

2007-11-25T18:46:00.000-08:00

Have you ever found a bug because, in response to some unexpected behavior on the part of the software under test, you starting "exploring" and tried some unplanned tests? Were your actions guided by your past experience in testing in that you had an intuition that there was a bug lurking in the code? When you found the bug did you feel guilty because the test that you performed was not part of the test plan? Did someone complain to you that "Hey - that test isn't in the plan!"

Well, you shouldn't have felt guilty. You were actually performing exploratory testing[1].

In an earlier post to this blog, I talked about how, in order to be effective, test plans have to be dynamic, not static, and have to adapt to changes, especially with regard to risks, in a project's development. But - it's impossible to anticipate in advance every test that will be necessary. Once you get your hands on the software to be tested, you often learn more about the product.

In exploratory testing, you combine your learning about the software under test and your test design with your test execution, all at the same time, as a single action and not as separate tasks. But wait, isn't this just ad hoc testing? The question about whether there is a real difference between ad hoc and exploratory testing is frequently raised. The best description[2][3] that I've seen about the substantive differences between these two types of testing is that while ad hoc testing tends to be random in nature, in exploratory testing, you rely on your testing experience to select paths that will uncover bugs, based on the behavior of the software under test. It's very "situational" in that the tests that you create are based on the multi-step situations that you create and the situations the software under test presents to you.

I once worked for a manager who practiced ad hoc testing in an extreme form. He frequently parachuted into projects, attempted a few tasks with the software under test, reported finding numerous bugs, and then walked away. However, the bugs that he reported tended to fall into two categories: legitimate bugs that there cosmetic or trivial, or user errors. In contrast, in exploratory testing, you're working more like a surgeon looking for a tumor. You've seen the signs of software tumors before (for example, degradation in performance caused by a memory leak) and you put that experience to use as you probe for high-value bugs. You start your testing with an idea of what tests you want to perform, and then based on situations caused by the software's behavior, coupled with your past experience, you start to explore complex situations. This situational testing is one of exploratory testing's strengths.

So, don't feel guilty. Observe the software under test, evaluate its behavior through the filter of your experience and keep on exploring!

Ref:

[1] Exploratory Testing Explained by James Bach

[2] http://blogs.msdn.com/imtesty/archive/2007/10/19/exploratory-testing-versus-ad-hoc-testing.aspx

[3] http://www.sqaforums.com/showflat.php?Cat=0&Board=UBB46&Number=406151&page=2&fpart=all

A Test Plan is a Tool - a Dynamic Tool, That is

2007-11-14T06:16:00.001-08:00

Why do we write test plans? Are we all just frustrated unpublished authors? (No, that's why we write blogs. ;-)

A test plan is a tool. The act of researching and writing a plan forces us to examine the software under test, to understand how it works, and the risks inherent it its function and its environment. By writing the plan as a document, we contribute to the institutional memory of the software project, as the document is persistent, and will be available as a resource to all the project team members and to other projects.

But - the fact that the plan may be stored as a static document file is purely incidental. Documents, files, database records are simply the mechanism by which we make collected information available to be read.

The most important thing about a test plan is that the information in the plan has to be dynamic, nit static. It has to adapt to changes in the project scope, direction, design, etc., and to changes in the potential risks that the project faces.

To be a successful test planning tool, the test plan has to reflect the "unfinished agenda" [1] of the testing effort.

But, what should that unfinished agenda be based on? The testing and quality risks that the product currently faces. The important thing to remember is that the risks the product faces during its testing will change. Perhaps the product design will change. If this happens, the plan has to adapt. Or, maybe new information will be received from a beta test sight about user requirements. Or, maybe tests that were originally planned become obsolete.

Remember how - in an earlier post to this blog - we talked about how it is impossible to find literally every bug in a product? The tests have to focus on what's most important, and that means focusing on what's most at risk. And - this will change constantly during the testing of a product.

So, the plan must be dynamic, not static, changing to meet the challenge of each new set of risks. The plan is part of a process - a way of finding the most serious bugs. [2]

Ref:

[1] From a 1960 speech by US President Kennedy
[2] Yes, from another JFK speech

What format should a test plan take? While the content is always more important than the format, it does help to have a well organized plan. I'm partial to the IEEE test plan format:

IEEE Test Plan
IEEE Test Plan in Wikipedia
Useful Wikipedia entry on test plans

The Goal of a SW Test is to Find a Bug

2007-11-08T07:19:00.000-08:00

"What's the goal of a software test?"

I always ask this question when I'm interviewing someone for a software development or test position. It sounds like an overly simplistic question, doesn't it? After all, we all know that:

The goal of software testing is to make sure there are no bugs, right?

Wrong! The goal of writing a software test is to find a bug.

This is why we test software. To locate the bugs and get them fixed. But, what about software that has no bugs?

There is no such thing as 100% "bug free" software. Why is this? The answer is inherent in the very nature of software. It's "soft." When you're working with physical media such as with steel, or concrete, or playdough, the limitations of what you can do are based on physical limitations of that media and the physical environment. With software, you face limitations of memory or CPU speed, but you are really only limited by your imagination. This is what makes software engineering so rewarding, and so much fun. You are basically building virtual structures out of ideas. And, unlike physical media, you can easily tear down, redesign, and rebuild structures is software easily. Sometimes badly. And so, there are bugs and you need new tests to find them.

But, hang on for a minute. In software testing, your goal cannot be to find literally every bug. You have to concentrate on finding the bugs that matter most. To do this requires an understanding of not only how the product under test works, the risks in its design, and how its customers will actually use it. And it requires that the tests that you write be intentionally destructive of and hostile to the product under test. It's often hard for people to be destructive of their own work. This is why you want an independent test team to create and execute tests.

Likewise, having a legacy library of thousands of tests that run cleanly does not guarantee that there are no bugs. It just means that the tests are not finding bugs. You have to constantly review the tests and map the test coverage to what risks the currently faces. Software products are dynamic. The test plans and tests have to adapt to keep pace.

In "The Art of Software Testing," Glenford Myers uses a medical analogy. If you feel ill and undergo medical tests that do not result in a diagnosis, is it accurate to say that the tests were "successful?" No! You're still sick, and your doctor hasn't run the correct test yet!

Ref: Glenford J. Myers, The Art of Software Testing. Wiley, 1979.

Ref: The goal of a software test: When failure equals success - IBM Developerworks article

Fundamentals, not "Philosophy"

2007-11-07T06:22:00.000-08:00

There's a great line in a book by Jack Nicklaus that I've always thought applies to software testing. When he was asked about his philosophy for approaching a difficult task (in his case, hitting a golf ball), his response was:

"I don't believe in philosophies. I believe in fundamentals."

I'll start a periodic series of posts on software testing fundamentals - "the rules" - soon.

Automated Open Source GUI Test Tool - Dogtail

2007-11-05T08:03:00.000-08:00

I came across this open source tool a while ago and was very impressed. The nice things about it are that it's open source, it's easy to get started using it, and, oh yes, it works!

The tool name is Dogtail - it's available: here

The technology it uses is interesting too. It uses Accessibility (A11Y) technologies to communicate with desktop applications. This is a key aspect of Dogtail's design. Unlike some other GUI test automation frameworks, Dogtail doesn't "scrape" information from the visual representation of the the application under the test's GUI into a data-store. Instead, it makes use of the accessibility-related metadata to create an in-memory model of the application's GUI elements.

I did not write Dogtail itself - I'm just a happy user. My contribution was some user documentation in the form of articles in Red Hat magazine. One of the article includes a Flash demo.

The articles are linked from the Dogtail entry in wikipedia

Introduction

2007-11-05T07:17:00.001-08:00

I’m starting this blog to keep track of useful software test tools and techniques that I find, and to relate my experiences in software engineering and testing. It can be a lonely feeling when you’re staring at a problem in testing software. I’m hoping that this blog can help...no, why do you ask? I’m sure that it’s fire-proof. Check the design spec. There must be some tests for that in the plan too...