Open Testware Reviews

GUI Test Driver Survey

Reviewed: 2003-May-30
Testingfaqs.org category: GUI Test Drivers

Testingfaqs.org gets a big boost for freeware GUI test tools this month. Seventeen freeware tools are now listed on the GUI Test Drivers list, rather than just the three we had before (Android, Jemmy, and Test Now).

Scope

GUI test tools are primarily functional test harnesses, like those on the "Test Drivers and Test Suite Management Tools" list, except that they specialize in automating a graphical user interface (GUI). Some GUI tools rely on other harnesses like JUnit, and focus on providing the mechanisms for faking mouse and keyboard input, similar to the way the commercial WinRunner tool interfaces with TestDirector.

About GUI testing

There has long been a dialog within the testing community about the difficulties of GUI test automation. Early GUI test tools were all "analog," which means that they record mouse movements using X-Y screen coordinates (which are measured digitally, but let's not pick nits :-) like those you used in school when you studied geometry. Analog capture and replay of test scripts is a very poor GUI test automation technique. They require a tremendous amount of maintenance every time the GUI design of the application under test changes, and in fact all of the test scripts may need to be recreated from scratch when then is a very minor change to the GUI. Analog scripts may also be sensitive to changes in screen resolution, color depth, and even where the window is placed on the screen. In most situations, it's better not to automate your GUI tests at all if your only choice is an analog tool.

Any modern GUI test tool worth its salt is "object-based," which means it can recognize many of the controls in a graphical application, like buttons, menus, and text input widgets, and can refer to them symbolically rather than with raw screen coordinates. This technique is much more resilient to changes in the GUI design, screen resolution, etc., though of course the tests will still need changes if a GUI control is added or deleted outright. A common limitation of an object-based tool is that it may have trouble automatically recognizing all of the controls in the application, especially if the developers use custom-developed controls or a toolkit that the tool isn't trained to understand. Object-based tools typically can also use analog-style screen coordinates if necessary.

One way to reduce the maintenance of a test case with analog-style scripting is to create a library that isolates all screen coordinate references to low-level functions. If the test cases only use these functions and never refer directly to screen coordinates, then when (not if) the application changes, most of the maintenance will be isolated to the test library. Still, that might represent significant effort. It may only be feasible to automate a small smoke test that's executed very frequently, rather than a large swath of functional tests.

There are some additional tool features that can further reduce test maintenance. One is "data-driven" testing. If you want to run the same test code repeatedly with only the test data varying from one run to the next, you can specify the data separately and have the script loop through each data value one at a time. Several commercial tools support this.

A more ambitious idea is "keyword-driven" testing. This involves specifying test data as with data-driven testing, but also uses pre-defined keywords to define actions for a test case to take. The keywords represent a very simple specification language that non-programmers may be able to use to directly develop automated tests. You still need automation engineers to implement the things that the keywords do, and with that comes all the usual issues of GUI automation. I know of two commercial tools that support keyword-driven testing - Certify from Worksoft, and the ABT Toolset (contact Hans Buwalda for a fact sheet). Hans also writes about this topic in his book Integrated Test Design and Automation: Using the Test Frame Method.

Comments on what you'll find here

Approach freeware GUI tools with caution. Looking at the documentation, it's very difficult to tell which of them support object-based automation. Building an analog GUI test tool is much easier than building an object-based tool, so I suspect that most of these tools are not object-aware.

Several of the tools are specific to Microsoft Windows - the documentation usually does not specify which Windows releases are supported. There are two here that are specific to one or more Unix variant. And there's the large subset of Java-based tools that presumably run across many platforms, but probably only for applications written in Java.

There are two freeware keyword-driven frameworks - EMOS Framework and SAFS, both only supported on Windows. Like their commercial counterparts, they are designed as wrappers on top of a limited selection of commercial test tools and can't easily stand on their own. You might hear about a data-driven tool called Robot DDE -- it has now been incorporated into SAFS.

Test Now is a library designed as an aid for another commercial test tool, Visual Test, though only for an outdated version of it. You can find a motley collection of other freeware code snippets for commercial GUI tools at the QA Downloads site.

A few tools have a graphical interface themselves, but most are implemented as libraries. Some can capture test scripts when you use the application under test, and the others require you to develop test scripts from scratch. A few of the Windows libraries are DLL's that can be used from many different languages. Several of the libraries are designed only for one language.

And now, the list

Tool	Platforms	Notes
Abbot	Java	GUI test library, with apparently some object-based recording capabilities
Android	Unix	Tcl/Tk extension for X11 testing, seems to be analog-only
AutoIt	Windows	Scripting language for analog GUI automation, also available as an ActiveX control and a DLL
EMOS Framework	Windows	Keyword-driven test framework for WinRunner, test cases are stored in spreadsheets
Jacareto	Java	Capture/replay tool.
Jemmy	Java	GUI test library for Java Swing/AWT
jfcUnit	Java	JUnit extension for testing Java Swing applications, with event recording
Marathon	Java	Python library for testing Java Swing applications
Perl Win32::GuiTest	Windows	GUI test library for Perl on Windows
Perl X11::GUITest	Linux/FreeBSD	GUI test library for Perl on X11
Phantom	Windows	GUI scripting language, newer tools based on Phantom are available commercially
Pounder	Java	Capture/replay tool
Ruby Win32::GuiTest	Windows	The beginnings of a port of Perl's Win32::GuiTest to Ruby
SAFS (Software Automation Framework Support)	Windows	Keyword-driven test framework for Rational Robot and WinRunner.
Test Now	Windows	Add-on library for Visual Test 4.0.
Tester	Windows	GUI test DLL with analog capture/replay
TRecorder	unknown	A capture/replay tool implemented in Delphi