Friday, June 19, 2009

Filling in the Hole in the UI Automation Tool Set

I've never been much of a UI test automation person as I've generally lived on the server-side of things. But, the past couple of years, I've found great set of open-source UI testing tools. In the past few days, I've found a tool that fills in what had been a gap in test tool coverage - automating Eclipse plugins.

The set of tools consists of:

For GNOME Applications - Dogtail

Dogtail developed by Zack Cerza and is written in Python (as are the tests that you write with it) and can be used to automate GNOME-based applications. The design of Dogtail is very interesting as uses accessibility technologies to communicate with desktop applications. It uses accessibility-related metadata to create an in-memory model of the application's GUI elements.

Dogtail can be found here: https://fedorahosted.org/dogtail

(There's also an entry in Wikipedia here: http://en.wikipedia.org/wiki/Dogtail that has links to the user documentation that I wrote for Dogtail. This was published as a series of articles in Red Hat Magazine http://magazine.redhat.com)

For Web Applications - Selenium

With Selenium, you can record or write test scripts that manipulate the web application to be tested through the browser. The tests can be written in HTML or Java or Python or other languages. There's also a record/playback mechanism.

Selenium can be found here: http://seleniumhq.org

For Eclipse plugins - SWTBot

I recently was able to fill in a long-empty hole in this set of tests when a co-worker of mine pointed me at a new tool named SWTBot. SWTBot is developed by Ketan Padegaonkar and automates Eclipse plugin testing.

SWTBot can be found here: http://www.eclipse.org/swtbot

SWTBot makes it very easy to build Java tests for Eclipse plugins. Here's a test code fragment that creates a new project:
SWTWorkbenchBot bot = new SWTWorkbenchBot();
bot.menu("File").menu("New").menu("Project...").click();
bot.tree().select("Java Project");
bot.button("Next >").click();
bot.textWithLabel("Project name:").setText("testProject");
bot.button("Finish").click();
SWTBot is very new; it's in the incubation phase of its development as an Eclipse project. The only problem that I've had with SWTBot so far is that it took me a little while to locate a set of example programs. I did find a good set here: http://code.google.com/p/swtbot-examples

So - to sum things up, with these tools, there's test coverage for desktop, web, and Eclipse-based apps. And that makes for a great tool-set!

Monday, June 15, 2009

Contributing to the Red Hat / JBoss SOA Platform Blog!

Just added a new post to the JBoss SOA Platform blog here:

http://jboss-soa-p.blogspot.com/2009/06/hanging-together-on-soa-platform.html

There's a feed for it here: http://www.jboss.org/feeds/view/soap?from=0

'Hoping to contribute to this blog on a regular basis going forward...


(No - you're not going blind - the same post - the subject is integrations - was in this blog until a few minutes ago. ;-)

Tuesday, May 12, 2009

grep - In living Color

In my last blog post, I mentioned that in performing software testing you tend to look in log files for clues as to the root cause of bugs. grep is a great tool for doing this, but it does have one drawback. If the lines of text in a log file that you are grep'ing through are very long, you can end up with output that looks like this:

grep here-is-the-bad.jar server.log

22:05:01,485 DEBUG [ServerInfo] class.path: /opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes:/usr/lib/here-is-the-bad.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes


Oh. Of course. There's the problem. What? You can't see it? ;-)

Let's try that grep command again. But this time, let's use grep's color option*. This will highlight what we're looking for:

grep --color=auto here-is-the-bad.jar server.log

22:05:01,485 DEBUG [ServerInfo] class.path: /opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes:/usr/lib/
here-is-the-bad.jar
:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxws.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-jaxrpc.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/server-saaj.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/serializer.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xercesImpl.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/jaxb-api.jar:/opt/local/system000001_test_1234/server-sample-p.1.2.4/server-as/lib/endorsed/xalan.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/resources.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/rt.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jsse.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/jce.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/lib/charsets.jar:/usr/lib/jvm/java-1.6.0-sun-1.6.0.7/jre/classes

Now, that's more like it! ;-)


* Special thanks to Ralph for pointing me at grep's color option!

Friday, May 8, 2009

Watching Things...with Linux's "watch"

When you test software, you tend to spend a lot of your waking hours (and some sleeping hours too) debugging problems. This involves looking around the system undertest for clues as to the root causes of failures.

One trick that I constantly use is to execute the "tail -f" command on system or server or process log files while I'm running tests. The tail command displays the last (10) lines of the file. Adding the -f option results in the file contents to be displayed as they are written to the log file. The result is that I can watch the log file as a real-time window into what's happening on the system under test.

I recently learned[1] about another useful tool; the "watch"[1] command. What watch does is to execute the shell command that you specify at the interval that you specify. What makes watch useful, though. is that is displays its output full-screen, and refreshes the full screen display, so you can don't have to scroll the display for the latest results.

For example, if you wanted to keep an eye on system memory use, you could execute this command to have watch display memory usage information and update the display every 5 seconds:

watch -n 5 cat /proc/meminfo

And see something like this:


Every 5.0s: cat /proc/meminfo Fri May 8 22:43:53 2009

MemTotal: 2049932 kB
MemFree: 948244 kB
Buffers: 77540 kB
Cached: 535776 kB
SwapCached: 0 kB
Active: 460124 kB
Inactive: 476236 kB
HighTotal: 1153728 kB
HighFree: 314396 kB
LowTotal: 896204 kB
LowFree: 633848 kB
SwapTotal: 2621432 kB
SwapFree: 2621432 kB
Dirty: 88 kB
Writeback: 0 kB
AnonPages: 323000 kB
Mapped: 67700 kB
Slab: 53548 kB
PageTables: 5064 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 3646396 kB
Committed_AS: 904128 kB
VmallocTotal: 114680 kB
VmallocUsed: 7748 kB
VmallocChunk: 106612 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 4096 kB

Or, if you wanted to only watch the amount of free memory, you could execute this variation on that same command:

watch -n 5 'cat /proc/meminfo | grep MemFree'

And see something like this:

Every 5.0s: cat /proc/meminfo | grep MemFree Fri May 8 22:47:10 2009

MemFree: 932252 kB

As I said, when you're debugging a test, or a potential bug uncovered by a test, you have to watch multiple source of information on the system you're testing. The watch command can be a very useful tool to help you see specific pieces of information in the flood of information that you may have to search through.

References:

[1] http://linux.about.com/library/cmd/blcmdl1_watch.htm

* Special thanks to Ralph for pointing me at watch!

Monday, March 30, 2009

Life as a Scientist

A great tool to analyze your blog: http://typealyzer.com/


(Found this tool referenced at Mark Little's blog: http://markclittle.blogspot.com/)

Saturday, March 21, 2009

Stress Testing: Don't spend your life waiting for a moment that just don't come, Well, don't waste your time waiting...

I remember the situation distinctly, even though it happened over a decade ago. The product in question was a 2nd generation server. The 1st generation was successful, but like many 1.0 projects, parts of it were, well, let's just say "immature." As a result, the product was buggy and difficult (and expensive!) to maintain. The goals of the 2nd generation product were ambitious; to correct the product's design flaws, extend its feature set, and improve its overall performance and reliability.

After a long development cycle, testing started and we were dismayed to find that the re-architecting effort had completely destabilized the product. We immediately started finding memory leaks, random crashes, loss of data, and basic functions that we just plain broken. The early test results caused a large amount of angst, and also caused us to adopt a very defensive posture. We reorganized testing into discrete stages, and we determined to not advance from one stage to the next until we had established a consistent baseline of reliable product operation.

As the early stages of functional testing progressed, product instability continued, and large numbers of bugs were found, resolved, and verified. Eventually, the product's stability improved, but there serious bugs were still periodically found, so we held to our defensive posture and did not progress from functional to system-wide or stress/performance tests until functional testing was complete. When we finally determined that the product was stable enough for stress testing, the project schedule had been extended multiple times. When stress testing started, began finding still more bugs related to product's inability to function under a load over time or to handle high levels of traffic. Once again, the project schedule had to be extended as large numbers of bugs were found, resolved, and verified.

What went wrong on this project? Well, quite a bit. The planning was overly optimistic, the design was not properly reviewed or understood, unit testing was inadequate, at one point, we even discovered that if every port on the server was physically cabled and the system was not properly rack-mounted, it would tip over and fall on the floor. In addition, by being too conservative in our test planning, and too rigid in our first reaction to the system's initial poor performance, we may have delayed the discovery of some serious problems.

What should we have done differently? We should have started some stress testing as early as could find situations and product configurations that would support it. The first stress tests could have taken the form of low levels of traffic running over an extended period of time, and not necessarily a massive system load at the start. By finding tests that the product could support, we could have established a baseline of performance to measure against as the product's many bugs were resolved (and its code was changed). By waiting for "perfect" conditions that actually arrived very late in the product's development, we delayed tests that could have been run under "good" conditions.

There's a saying that, "if you see blue sky, go for it."[1] It can be like that in software testing too. If you wait for perfect conditions to run some tests, you may end up waiting for a long time, and that waiting may result in bugs being found at a late and probably inconvenient time. When should you try stress testing a product? As early as you can. Performance related problems can be difficult and time consuming to resolve. The sooner you find them, the better. So, if you see some blue sky among the bugs early in a project test cycle, go for it...

(Special thanks to Pavel!...and Mr. Springsteen...)

References:

[1] http://www.pbs.org/wgbh/amex/kennedys/filmmore/pt.html

Monday, February 16, 2009

Parameterized Testing? But wait, there's more! (Inversion of Control/Dependency Injection, that is)

When you spend most of your waking hours (and, truth be told, some of you sleeping hours too) involved in software engineering, you can find yourself speaking a language foreign to friends and relatives. In my own case, most of my friends and family have no experience in or with software design, development, or testing. They tend to see my software engineering experience as useful, but also something out of the ordinary.

This past Monday, for example, a good friend of mine, a lawyer by profession, telephoned me late at night, desparate to rescue his home computer from the malware that he had inadvertently downloaded. I managed to walk him through the steps necessary to get his computer working again. That is, I thought I had, until a couple of nights later, he called me again asking my advice in buying a new computer.

A few days later, I was reading a printed copy of Martin Fowler's paper "Inversion of Control Containers and the Dependency Injection pattern" [1] when a friend of mine commented, "inversion of control?" Dude, that's the story of my life. My kids rule the house."

The term "dependency injection" can at first be bit more intimidating than the reality of what it is and how it's used. The simplest explanation that I've ever seen for it is in James Shore's excellent blog "The Art of Agile" [2] when he states that:

'...dependency injection means giving an object its instance variables..'

OK. This is all interesting, but what does it have to do with software testing? Quite a bit, actually. In the blog entry from a few weeks ago titled "Parameterized Testing - Very Easy with TestNG's DataProvider" we talked about designing tests that accept parameters, so that the same test could be run against multiple sets of data. In this post, we'll revisit that test design pattern.

But first, let's take a deeper look at inversion of control and dependency injection.

To begin, let's disect the phrase "inversion of control." In this context, who wants control, and just how is this control inverted?

In an object oriented language such as Java, you solve programming problems by designing classes. The classes contain variables (to hold data) and methods (functions that manipulate the data). To actually perform the tasks that you want the programs to do, you create discrete instances of the objects. When you create an object instance, you work with instances of the variables.

Let's say that you've defined a Java class of type Car, and within the class you set a Manufacturer variable to, say, "Audi."

What have we just done in terms of control and dependencies?

* Who has control? The "Car" class has the control. In order to build a new car, it defines a variable of type "Manufacturer." Or more precisely it defines an instance variable of type "Manufacturer." In this case, the class only allows for a Manufacturer of "Audi."

* The "Car" class needs the "Manufacturer" variable to do its work. In other words, it depends on this variable. That's right, this instance variable is a "dependency."

However, there's a problem with the "Car" class. How can build a test for the class, and test multiple car manufacturers? To build a test for Toyota, for example, we will have to build another class that's almost identical to the "Car" class, except that it will create a different "Manufacturer" object.

The answer is easy, of course, we just add another constructor to the Car class:

public Car(Manufacturer targetManufacturer) {
carManufacturer = targetManufacturer;
}

So, now what have we just done in terms of control and dependencies?

* The value of the dependency, that is, the "carManufacturer" variable is not hardwired into the "Car" class. It is "injected" into the "Car" class through its new constructor.

* Who has control now? Not the "Car" class anymore. The class can now be used to create any type of car. Who controls the decision of the type of car to create? The program (or test program) that creates the instance of the "Car" class. That's the "inversion" of control.

One important use of this model is that it emables us to isolate the class being tested. For example, if we want to test the Car class with a mock or stub[3] version of the Manufacturer class, we can pass that stub or mock object to the Car class during the test. This will let the test concentrate on verifying the operation of the Car class, independent of the function (or lack of function) provided by the Manufacturer class.

When you couple this design pattern with ability to pass a parameter into a test, such as is supported in TestNG, this enables us to inject the dependecies into the classes being tested. This makes it possible for the tests to control not only the test data, but also the defined profile and characteristics of the classes under test. And to do it without hardcoding data into the tests or the classes under test. Now, that's control!

References:

[1] http://martinfowler.com/articles/injection.html

[2] http://jamesshore.com/Blog/Dependency-Injection-Demystified.html

[3] Next Generation Java Testing by Cedric Beust and Hani Suleiman, pages 95-96. http://testng.org/doc/book.html