Riftsaw - Open Source BPEL!

I just started posting to another blog - this one is for the "Riftsaw" project. Riftsaw is JBoss's open source BPEL engine. Very cool stuff...

http://riftsaw.blogspot.com/2009/10/impatient-start-with-jboss-riftsaw-5.html

http://riftsaw.blogspot.com/2009/11/from-bpel-to-esb-and-back-introduction.html

http://riftsaw.blogspot.com/2009/11/from-esb-to-bpel-continuing-with.html

Shocked by the Disk Space Used by iPhoto - When QE Experience Comes in Handy

I was backing up by iPhoto library today and was shocked to see that had recently grown in size by about 6GB. This increase in disk usage happened while I had added no more than (100) or so new pictures.

Huh?

At first, I assumed that some form of corruption had happened. On closer inspection, however, I found the true culprit:

du -h | grep G
6.0G    ./iPod Photo Cache
1.0G    ./Modified/2007
2.6G    ./Modified
2.0G    ./Originals/2007
1.6G    ./Originals/2008
1.2G    ./Originals/2009
6.8G    ./Originals
16G    .

Nothing had happened to the iPhoto library or database. The increase in disk space usage was due to my finally getting an iPod that could handle pictures. ;-)

But - there's a lesson here that all software QE engineers have learned. When you think you've found a new bug, you have to examine the software under test for any changes, AND you also have to examine the environment in which the software runs for changes.

I like to look at software testing as an exercise in algebra. Your task is to take a very complex equation and find its answer. In that equation, you have some variables to resolve. The trick some times is that the variables may not be so obvious at first! Even if they are 6GB in size...

Another DZone post!

DZone just picked up the Byteman post - it's here:

http://java.dzone.com/news/fault-injection-testing-first?mz=3006-jboss

Thinking about Scrum and Base Running

I attended an excellent class on Scrum[1] a couple of weeks ago. I've always been a bit of a skeptic of formal project management or development frameworks, but I have to say that I was impressed. Some elements of scrum are really either common sense, such as the need for transparency, and others, such as short development cycles that result in an always functioning system hold real promise.

The one aspect of scrum that I identified most with, however, was "discover and adapt." In order to be successful in testing software, you always have to operating in a discover and adapt mode. You can define a detailed test plan, and base that plan on the best information available at the time, but in the course of a software project test cycle, you have to constantly re-evaluate the current state of the software under test, and then adapt your future plans to match. You may start off by testing subsystem A, but then find that its testing is blocked, but you can continue to make progress by testing subsystem B.

One crucial part of your adapting to what you discover about the software under test, is that your priorities for testing will always change. You highest priority will always be to run tests that will find the most serious as yet undiscovered bug. But, the places in the code where that bug may be lurking can change as either the code changes, or your understanding of the code increases.

There's a great analogy for "discover and adapt" in sports. (And, no, it doesn't involve golf this time. ;-) It involves American baseball, and concerns the sometimes under appreciated skill of base running.

When you think about what it takes to be a great base runner, you might think that the only factor is speed. Speed is important, to be sure, but what is even more important is the base runner's judgment. He has to be able to adapt quickly to changing situations, and decide when to be aggressive and "take the extra base," and when to be more conservative. In short, he has to discover and adapt. How does he do this? He watches the ball and the opposing players, not the bases. The bases aren't changing, the position of the ball and those players is.

I found this wonderful passage about Joe DiMaggio in David Halberstam's book[2] "Summer of '49" that describes this:

...Stengel, his new manager, was equally impressed, and when DiMaggio was on base he would point to him as an example of the perfect base runner. "Look at him," Stengel would say as DiMaggio ran out a base hit, "he's always watching the ball. He isn't watching second base. He isn't watching third base. He knows they haven't been moved. He isn't watching the ground, because he knows they haven't built a canal or a swimming pool since he was last there. He's watching the ball and the outfielder, which is the one thing that is different on every play..."

It's like that in software too. You can't keep your eye on static plans, you have to keep your eye on the current (and changing) state of the project and the code, and always discover and adapt.

References

[1] http://en.wikipedia.org/wiki/Scrum_(development)

[2] Halberstam, David, "Summer of '49,", (Morrow: New York), 1989. http://www.amazon.com/Summer-49-P-S-David-Halberstam/dp/0060884266/ref=sr_1_1?ie=UTF8&s=books&qid=1255145843&sr=8-1

Fault Injection Testing - First Steps with JBoss Byteman

Fault injection testing[1] is a very useful element of a comprehensive test strategy in that it enables you to concentrate on an area that can be difficult to test; the manner in which the application under test is able to handle exceptions.

It's always possible to perform exception testing in a black box mode, where you set up external conditions that will cause the application to fail, and then observe those application failures. Setting and automating (and reproducing) these such as these can, however, be time consuming. (And a pain in the neck, too!)

JBoss Byteman

I recently found a bytecode injection tool that makes it possible to automate fault injection tests. JBoss Byteman[2] is an open-source project that lets you write scripts in a Java-like syntax to insert events, exceptions, etc. into application code.

Byteman version 1.1.0 is available for download from: http://www.jboss.org/byteman - the download includes a programmer's guide. There's also a user forum for asking questions here: http://www.jboss.org/index.html?module=bb&op=viewforum&f=310, and a jboss.org JIRA project for submitted issues and feature requests here: https://jira.jboss.org/jira/browse/BYTEMAN

A Simple Example

The remainder of this post describes a simple example, on the scale of the classic "hello world" example, of using Byteman to insert an exception into a running application.

Let's start by defining the exception that we will inject into our application:

1  package sample.byteman.test;
2
3  /**
4   * Simple exception class to demonstrate fault injection with byteman
5   */
6
7  public class ApplicationException extends Exception {
8
9    private static final long serialVersionUID = 1L;
10   private int intError;
11   private String theMessage = "hello exception - default string";
12
13   public ApplicationException(int intErrNo, String exString) {
14       intError = intErrNo;
15       theMessage = exString;
16   }
17
18   public String toString() {
19       return "**********ApplicationException[" + intError + " " + theMessage + "]**********";
20   }
21
22  } /* class */

There's nothing complicated here, but note the string that is passed to the exception constructor at line 13.

Now, let's define our application class:

1  package sample.byteman.test;
2
3  /**
4   * Simple class to demonstrate fault injection with byteman
5   */
6
7  public class ExceptionTest {
8
9      public void doSomething(int counter) throws ApplicationException {
10        System.out.println("called doSomething(" + counter + ")");
11            if (counter > 10) {
12               throw new ApplicationException(counter, "bye!");
13            }
14            System.out.println("Exiting method normally...");
15    } /* doSomething() */
16
17    public static void main(String[] args) {
18        ExceptionTest theTest = new ExceptionTest();
19        try {
20            for (int i = 0; i < 12; i ++) {
21                theTest.doSomething (i);
22            }
23        } catch (ApplicationException e) {
24            System.out.println("caught ApplicationException: " + e);
25        }
26    }
27
28 } /* class*/

The application instantiates an instance of ExceptionTest at line 18, then runs the doSomething method in a loop until a counter is greater then 10. Then it raises the exception that we defined earlier. When we run the application, we see this output:

java -classpath bytemanTest.jar sample.byteman.test.ExceptionTest
called doSomething(0)
Exiting method normally...
called doSomething(1)
Exiting method normally...
called doSomething(2)
Exiting method normally...
called doSomething(3)
Exiting method normally...
called doSomething(4)
Exiting method normally...
called doSomething(5)
Exiting method normally...
called doSomething(6)
Exiting method normally...
called doSomething(7)
Exiting method normally...
called doSomething(8)
Exiting method normally...
called doSomething(9)
Exiting method normally...
called doSomething(10)
Exiting method normally...
called doSomething(11)
caught ApplicationException: **********ApplicationException[11 bye!]**********

OK. Nothing too exciting so far. Let's make things more interesting by scripting a Byteman rule to inject an exception before the doSomething method has a chance to print any output. Our Byteman script looks like this:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example - throw an exception
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND buffer = 0
9   IF TRUE
10     DO throw sample.byteman.test.ApplicationException(1,"ha! byteman was here!")
11  ENDRULE

• Line 4 - RULE defines the start of the RULE. The following text on this line is not executed
  
• Line 5 - Reference to the class of the application to receive the injection
  
• Line 6 - And the method in that class. Note that since if we had written this line as "METHOD doSomething", the rule would have matched any signature of the soSomething method
  
• Line 7 - Our rule will fire when the PrintStream.println method is invoked
  
• Line 8 - BIND determince values for variables which can be referenced in the rule body - in our example, the recipient of the doSomething method call that triggered the rule, is identified by the parameter reference $0
  
• Line 9 - A rule has to include an IF clause - in our example, it's always true
  
• Line 10 - When the rule is triggered, we throw an exception - note that we supply a string to the exception constructor
  

Now, before we try to run this run, we should check the its syntax. To do this, we build our application into a .jar (bytemanTest.jar in our case) and use bytemancheck.sh

sh bytemancheck.sh -cp bytemanTest.jar byteman.txt
checking rules in sample_byteman.txt
TestScript: parsed rule Simple byteman example - throw an exception
RULE Simple byteman example - throw an exception
CLASS sample.byteman.test.ExceptionTest
METHOD doSomething(int)
AT INVOKE PrintStream.println
BIND buffer : int = 0
IF   TRUE
DO   throw (1"ha! byteman was here!")

TestScript: checking rule Simple byteman example - throw an exception
TestScript: type checked rule Simple byteman example - throw an exception

TestScript: no errors

Once we get a clean result, we can run the application with Byteman. To do this, we run the application and specify an extra argument to the java command. Note that Byteman requires JDK 1.6 or newer.

java -javaagent:/opt/Byteman_1_1_0/build/lib/byteman.jar=script:sample_byteman.txt -classpath bytemanTest.jar sample.byteman.test.ExceptionTest

And the result is:

caught ApplicationException: **********ApplicationException[1 ha! byteman was here!]**********

Now that the Script Works, Let's Improve it!

Let's take a closer look and how we BIND to a method parameter. If we change the script to read as follows:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example - throw an exception
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND counter = $1
9   IF TRUE
10   DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!")
11   ENDRULE

In line 8, the BIND clause now refers to the int method parameter by index using the syntax $1. This change makes the value available inside the rule body by enabling us to use the name "counter." The value of counter is then supplied as the argument to the constructor for the ApplicationException class. This new version of the rule demonstrates shows how we can use local state as derived from the trigger method
to construct our exception object.

But wait there's more! Let's use the "counter" value as a counter.

It's useful to be able to force an exception the first time a method is called. But, it's even more useful to be able to force an exception at a selected invocation of a method. Let's add a test for that counter value to the script:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example 2 - throw an exception at 3rd call
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND counter = $1
9   IF counter == 3
10   DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!")
11   ENDRULE

In line 9, we've changed the IF clause to make use of the counter value. When we run the test with this script, the first 2 calls to doSomething succeed, but the third one fails.



One Last Thing - Changing the Script for a Running Process

So far, so good. We've been able to inject a fault/exception into our running application, and even specify which iteration of a loop in which it happens. Suppose, however, we want to change a value in a byteman script, while the application is running? No problem! Here's how.

First, we need to alter our application so that it can run for a long enough time for us to alter the byteman script. Here's a modified version of the doSomething method that waits for user input:

1    public void doSomething(int counter) throws ApplicationException {
2       
3    BufferedReader lineOfText = new BufferedReader(new InputStreamReader(System.in));
4        try {
5            System.out.println("Press <return>");
6            String textLine = lineOfText.readLine();
7        } catch (IOException e) {
8            e.printStackTrace();
9        }
10       
11       System.out.println("called doSomething(" + counter + ")");
12       if (counter > 10) {
13           throw new ApplicationException(counter, "bye!");
14       }
15       System.out.println("Exiting method normally...");
16   } 

If we run this version of the application, we'll see output like this:

Press <return>

called doSomething(0)
Exiting method normally...
Press <return>

called doSomething(1)
Exiting method normally...
Press <return>

called doSomething(2)
Exiting method normally...
caught ApplicationException: **********ApplicationException[3 ha! byteman was here!]**********  

Let's run the application again, but this time, don't press <return>. While the application is waiting for input, create a copy of the byteman script. In this copy, change the IF clause to have a loop counter set to a different value, say '5.' Then, open up a second command shell window and enter this command:

Byteman_1_1_0/bin/submit.sh sample_byteman_changed.txt

Then, return to the first command shell window and start pressing return, and you'll see this output:

Press <return>
redefining rule Simple byteman example - throw an exception

called doSomething(0)
Exiting method normally...
Press <return>

called doSomething(1)
Exiting method normally...
Press <return>

called doSomething(2)
Exiting method normally...
Press <return>

called doSomething(3)
Exiting method normally...
Press <return>

called doSomething(4)
Exiting method normally...
caught ApplicationException: **********ApplicationException[5 ha! byteman was here!]**********  

So, we were able to alter the value in the original byteman script, without stopping the application under test!


Pitfalls Along the Way

Some of the newbee mistakes that I made along the way were:

• Each RULE needs an IF clause - even if you want the rule to always fire
  
• The methods referenced in a RULE cannot be static - if they are static, then there is no $0 (aka this) to reference
  
• Yes, I had several errors and some typos the first few times I tried this. A syntax checker is always my best friend. ;-)
  

Closing Thoughts

With this simple example, we're able to inject injections into a running application in an easily automated/scripted manner. But, We've only scratched the surface with Byteman. In subsequent posts, I'm hoping to explore using Byteman to cause more widespread havoc in software testing.

References

[1] http://en.wikipedia.org/wiki/Fault_injection

[2] http://www.jboss.org/byteman

(Special thanks to Andrew Dinn for his help! ;-)

Christmas? Already????

The stores are already setting up for Christmas. This is very depressing as it is only October.

But - it's never too early for some excellent Christmas music. Check this out:



Christmas Music by Magnatune Compilation

New Post - in a new Blog!

I just posted to a new blog - the JBoss ESB project blog:

http://jbossesb.blogspot.com/2009/10/content-based-routing-in-jbossesb-just.html