Fault Injection Testing - First Steps with JBoss Byteman

Fault injection testing[1] is a very useful element of a comprehensive test strategy in that it enables you to concentrate on an area that can be difficult to test; the manner in which the application under test is able to handle exceptions.

It's always possible to perform exception testing in a black box mode, where you set up external conditions that will cause the application to fail, and then observe those application failures. Setting and automating (and reproducing) these such as these can, however, be time consuming. (And a pain in the neck, too!)

JBoss Byteman

I recently found a bytecode injection tool that makes it possible to automate fault injection tests. JBoss Byteman[2] is an open-source project that lets you write scripts in a Java-like syntax to insert events, exceptions, etc. into application code.

Byteman version 1.1.0 is available for download from: http://www.jboss.org/byteman - the download includes a programmer's guide. There's also a user forum for asking questions here: http://www.jboss.org/index.html?module=bb&op=viewforum&f=310, and a jboss.org JIRA project for submitted issues and feature requests here: https://jira.jboss.org/jira/browse/BYTEMAN

A Simple Example

The remainder of this post describes a simple example, on the scale of the classic "hello world" example, of using Byteman to insert an exception into a running application.

Let's start by defining the exception that we will inject into our application:

1  package sample.byteman.test;
2
3  /**
4   * Simple exception class to demonstrate fault injection with byteman
5   */
6
7  public class ApplicationException extends Exception {
8
9    private static final long serialVersionUID = 1L;
10   private int intError;
11   private String theMessage = "hello exception - default string";
12
13   public ApplicationException(int intErrNo, String exString) {
14       intError = intErrNo;
15       theMessage = exString;
16   }
17
18   public String toString() {
19       return "**********ApplicationException[" + intError + " " + theMessage + "]**********";
20   }
21
22  } /* class */

There's nothing complicated here, but note the string that is passed to the exception constructor at line 13.

Now, let's define our application class:

1  package sample.byteman.test;
2
3  /**
4   * Simple class to demonstrate fault injection with byteman
5   */
6
7  public class ExceptionTest {
8
9      public void doSomething(int counter) throws ApplicationException {
10        System.out.println("called doSomething(" + counter + ")");
11            if (counter > 10) {
12               throw new ApplicationException(counter, "bye!");
13            }
14            System.out.println("Exiting method normally...");
15    } /* doSomething() */
16
17    public static void main(String[] args) {
18        ExceptionTest theTest = new ExceptionTest();
19        try {
20            for (int i = 0; i < 12; i ++) {
21                theTest.doSomething (i);
22            }
23        } catch (ApplicationException e) {
24            System.out.println("caught ApplicationException: " + e);
25        }
26    }
27
28 } /* class*/

The application instantiates an instance of ExceptionTest at line 18, then runs the doSomething method in a loop until a counter is greater then 10. Then it raises the exception that we defined earlier. When we run the application, we see this output:

java -classpath bytemanTest.jar sample.byteman.test.ExceptionTest
called doSomething(0)
Exiting method normally...
called doSomething(1)
Exiting method normally...
called doSomething(2)
Exiting method normally...
called doSomething(3)
Exiting method normally...
called doSomething(4)
Exiting method normally...
called doSomething(5)
Exiting method normally...
called doSomething(6)
Exiting method normally...
called doSomething(7)
Exiting method normally...
called doSomething(8)
Exiting method normally...
called doSomething(9)
Exiting method normally...
called doSomething(10)
Exiting method normally...
called doSomething(11)
caught ApplicationException: **********ApplicationException[11 bye!]**********

OK. Nothing too exciting so far. Let's make things more interesting by scripting a Byteman rule to inject an exception before the doSomething method has a chance to print any output. Our Byteman script looks like this:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example - throw an exception
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND buffer = 0
9   IF TRUE
10     DO throw sample.byteman.test.ApplicationException(1,"ha! byteman was here!")
11  ENDRULE

• Line 4 - RULE defines the start of the RULE. The following text on this line is not executed
  
• Line 5 - Reference to the class of the application to receive the injection
  
• Line 6 - And the method in that class. Note that since if we had written this line as "METHOD doSomething", the rule would have matched any signature of the soSomething method
  
• Line 7 - Our rule will fire when the PrintStream.println method is invoked
  
• Line 8 - BIND determince values for variables which can be referenced in the rule body - in our example, the recipient of the doSomething method call that triggered the rule, is identified by the parameter reference $0
  
• Line 9 - A rule has to include an IF clause - in our example, it's always true
  
• Line 10 - When the rule is triggered, we throw an exception - note that we supply a string to the exception constructor
  

Now, before we try to run this run, we should check the its syntax. To do this, we build our application into a .jar (bytemanTest.jar in our case) and use bytemancheck.sh

sh bytemancheck.sh -cp bytemanTest.jar byteman.txt
checking rules in sample_byteman.txt
TestScript: parsed rule Simple byteman example - throw an exception
RULE Simple byteman example - throw an exception
CLASS sample.byteman.test.ExceptionTest
METHOD doSomething(int)
AT INVOKE PrintStream.println
BIND buffer : int = 0
IF   TRUE
DO   throw (1"ha! byteman was here!")

TestScript: checking rule Simple byteman example - throw an exception
TestScript: type checked rule Simple byteman example - throw an exception

TestScript: no errors

Once we get a clean result, we can run the application with Byteman. To do this, we run the application and specify an extra argument to the java command. Note that Byteman requires JDK 1.6 or newer.

java -javaagent:/opt/Byteman_1_1_0/build/lib/byteman.jar=script:sample_byteman.txt -classpath bytemanTest.jar sample.byteman.test.ExceptionTest

And the result is:

caught ApplicationException: **********ApplicationException[1 ha! byteman was here!]**********

Now that the Script Works, Let's Improve it!

Let's take a closer look and how we BIND to a method parameter. If we change the script to read as follows:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example - throw an exception
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND counter = $1
9   IF TRUE
10   DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!")
11   ENDRULE

In line 8, the BIND clause now refers to the int method parameter by index using the syntax $1. This change makes the value available inside the rule body by enabling us to use the name "counter." The value of counter is then supplied as the argument to the constructor for the ApplicationException class. This new version of the rule demonstrates shows how we can use local state as derived from the trigger method
to construct our exception object.

But wait there's more! Let's use the "counter" value as a counter.

It's useful to be able to force an exception the first time a method is called. But, it's even more useful to be able to force an exception at a selected invocation of a method. Let's add a test for that counter value to the script:

1   #
2   # A simple script to demonstrate fault injection with byteman
3   #
4   RULE Simple byteman example 2 - throw an exception at 3rd call
5   CLASS sample.byteman.test.ExceptionTest
6   METHOD doSomething(int)
7   AT INVOKE PrintStream.println
8   BIND counter = $1
9   IF counter == 3
10   DO throw sample.byteman.test.ApplicationException(counter,"ha! byteman was here!")
11   ENDRULE

In line 9, we've changed the IF clause to make use of the counter value. When we run the test with this script, the first 2 calls to doSomething succeed, but the third one fails.



One Last Thing - Changing the Script for a Running Process

So far, so good. We've been able to inject a fault/exception into our running application, and even specify which iteration of a loop in which it happens. Suppose, however, we want to change a value in a byteman script, while the application is running? No problem! Here's how.

First, we need to alter our application so that it can run for a long enough time for us to alter the byteman script. Here's a modified version of the doSomething method that waits for user input:

1    public void doSomething(int counter) throws ApplicationException {
2       
3    BufferedReader lineOfText = new BufferedReader(new InputStreamReader(System.in));
4        try {
5            System.out.println("Press <return>");
6            String textLine = lineOfText.readLine();
7        } catch (IOException e) {
8            e.printStackTrace();
9        }
10       
11       System.out.println("called doSomething(" + counter + ")");
12       if (counter > 10) {
13           throw new ApplicationException(counter, "bye!");
14       }
15       System.out.println("Exiting method normally...");
16   } 

If we run this version of the application, we'll see output like this:

Press <return>

called doSomething(0)
Exiting method normally...
Press <return>

called doSomething(1)
Exiting method normally...
Press <return>

called doSomething(2)
Exiting method normally...
caught ApplicationException: **********ApplicationException[3 ha! byteman was here!]**********  

Let's run the application again, but this time, don't press <return>. While the application is waiting for input, create a copy of the byteman script. In this copy, change the IF clause to have a loop counter set to a different value, say '5.' Then, open up a second command shell window and enter this command:

Byteman_1_1_0/bin/submit.sh sample_byteman_changed.txt

Then, return to the first command shell window and start pressing return, and you'll see this output:

Press <return>
redefining rule Simple byteman example - throw an exception

called doSomething(0)
Exiting method normally...
Press <return>

called doSomething(1)
Exiting method normally...
Press <return>

called doSomething(2)
Exiting method normally...
Press <return>

called doSomething(3)
Exiting method normally...
Press <return>

called doSomething(4)
Exiting method normally...
caught ApplicationException: **********ApplicationException[5 ha! byteman was here!]**********  

So, we were able to alter the value in the original byteman script, without stopping the application under test!


Pitfalls Along the Way

Some of the newbee mistakes that I made along the way were:

• Each RULE needs an IF clause - even if you want the rule to always fire
  
• The methods referenced in a RULE cannot be static - if they are static, then there is no $0 (aka this) to reference
  
• Yes, I had several errors and some typos the first few times I tried this. A syntax checker is always my best friend. ;-)
  

Closing Thoughts

With this simple example, we're able to inject injections into a running application in an easily automated/scripted manner. But, We've only scratched the surface with Byteman. In subsequent posts, I'm hoping to explore using Byteman to cause more widespread havoc in software testing.

References

[1] http://en.wikipedia.org/wiki/Fault_injection

[2] http://www.jboss.org/byteman

(Special thanks to Andrew Dinn for his help! ;-)