Toward a Standard Test Harness

Software QA Magazine, Vol. 3, No. 2, April 30 1996 Tester's Toolbox column

by Danny Faught
Copyright 1996, Danny Faught

Is there a standard test suite structure?  Perhaps the best contender
is the Test Environment Toolkit, or "TET".  I cast my net out on the
Internet to learn both the facts and opinions about TET.  I want to
thank everyone who offered their insights.  In this column,
I'll explore TET's industry acceptance, and wander down some
interesting alleys in the process.

In 1989, X/Open, the Open Software Foundation, and UNIX International
combined forces to kick off the TET project.  Their goal was "to define
a common interface for the development and use of conformance tests."
[From http://tetworks.xopen.org/tet.htm]  They involved the rest of
the software testing community by inviting others to participate in the
requirements definition and by making the implementation freely available.

After the requirements were defined, the goal became more aggressive:
"...the project now expects that a substantial portion of the test
development community will begin using TET for the development
of conformance testing software."  [Test Environment Toolkit
Programmer's Guide v1.7, p. 1-1]  Now the claim is that TET is the
"single most broadly used framework for test development."  [From
http://tetworks.xopen.org/tet.htm]  I didn't find any solid proof
of that, but there is some anecdotal evidence supporting the claim.
While TET may have the largest installed base, I don't think any test
harness is pervasive at this point.

The two main components of TET are the Test Case Controller (TCC) and
the Test Case Manager (TCM).  The TCC is the execution manager which
the user runs from the command line.  The TCM is the application
programming interface (API) that controls things from the test's point
of view.  The base TET distribution includes APIs for C and the XPG
shell.  I'll use "test harness" as a generic term for tools like the
TCC/TCM duo.


Room for improvement
--------------------

One complaint I heard in several places was that TET lacked support.
Apparently, support for TET has not been strong for the last few
years.  It seems that the need for support is more for enhancements
than bug fixes.  People who were familiar with the large volunteer
support base for perl, or the dedicated support provided by the
developer of expect, couldn't understand why TET didn't develop a
similar level of support.  Individual groups would maintain their own
patches that they would apply to each new version of TET they
received.

Another development that many people think has weakened TET's story is
the appearance of two TET derivatives.  ETET is an extension to TET
developed by SunSoft and revised by USL.  ETET adds APIs for C++, perl,
and ksh, and adds several features that enhance usability.  dTET was
developed by X/Open itself to use a heterogeneous client/server
architecture to address the needs of networking tests.  Both of the
extensions followed the original TET specification, and they are
capable of running tests designed for the base TET implementation.

Some see the fragmentation as a major problem because they don't know
which flavor is the right one to use.  However, X/Open considers
it an asset.  They encourage people to experiment with their own 
enhancements to the bare-bones TET.  

Recent announcements by X/Open address some of these concerns.  One is
the "Revised TET" project, which will bring ETET and dTET functionality
back into the base TET implementation.  Any compatibility issues will
be handled by a compatibility option, so it should be easy to move
existing tests to the revised TET.  Also, the list of supported
platforms will be expanded from just Unix and POSIX-compliant systems to
include Windows NT, OS/2, and DEC Alpha running Open VMS.  Remote-only
support will be added for MS-Windows 3.11 and Apple PowerMac System 7.
The plan is to have the new version of TET available by the end of the
year.

Another recent announcement details a new support model.  X/Open now
offers TET support for an annual fee of $10,000.  Supported customers
will be given priority treatment for bug reporting and will get access
to maintenance releases six months earlier than the general public.
Users who don't ante up will still have access to the new releases for
Unix after the six month wait.


Should a test harness use an API?
---------------------------------

"Because of the overriding concern for providing flexibility for the
test developer, the Toolkit imposes a minimum amount of structure
on test cases," the TET documentation says [Test Environment Toolkit
Architectural, Functional, and Interface Specification, page 2].  But the
influence of the test case manager on the code seems to be pervasive.
For example, using the C API, you don't even have control over the
main() function.  Calls to the API are scattered through the code.
I'm not sure what part of the structure is minimized.

It's interesting to compare this framework to HP-Convex's home-grown test
environment, the Convex Integrated Test Environment (CITE).  [Peter Vogel,
"An Integrated General Purpose Automated Test Environment", Proceedings
of the 1993 International Symposium on Software Testing and Analysis,
pp. 61-69.  Comparison with TET on pp.  63-64.]  CITE uses a rule language
rather than an API to control the flow of a test and to determine the
test's results.  The interface between the test harness and the test
executable is very clean.  And they can use any language at all for
the test executable, without worrying whether the API is available in
that language.  They've incorporated tests from many sources under CITE
with little or no changes to the source.

The tradeoff between rule-based and API-based test drivers is an
interesting one.  I think that the benefit of taking the effort to use
TET's API is that the tests are more structured, and more likely to
follow a common form.

A testament to TET's flexibility is the fact that you can use the API
to build a wrapper that works similarly to CITE's rule language.  It
seems to be a common practice to move test functionality to an external
process when the needs of a test program aren't quite met by the test
harness.  I've done this with CITE as well in some cases.  Or, you can
use the TET_OUTPUT_CAPTURE option, which was designed to support tests
that were not written using the API.


Assertion-based
---------------

TET is used widely as a test harness for conformance tests.  But it
seems to get a lot more resistance in other testing arenas.  It uses an
"assertion-based" approach, which works best for simple tests like
conformance tests.  Other types of tests, such as stress tests, are
more complex than a few setup procedures and a simple assertion.

I haven't been able to dig into all the issues that people have with
broadening TET's scope beyond conformance testing.  I imagine that
a number of the organizations currently using TET aren't just doing
conformance testing with it.  Maybe in some cases, people are making
demands on TET that could be solved externally.  Using CITE, I've used
a number of specialized intermediate drivers that run under the harness, 
and they translate the results to a form that the harness expects.
The high-level assertions are conceptually a part of the rule language.


PureTestExpert
--------------

A test harness offered by Pure Software has helped to establish TET's
foothold.  Pure does not tout the fact that PureTestExpert uses TET;
they simply refer to an "Execution Engine".  But the tool is full of
TET artifacts, and Pure confirms that it indeed incorporates TET.
Future versions may add support for other frameworks as well.

PureTestExpert provides a full graphical interface on top of TET.  It
also adds many other features, such as a test documentation database, a
test results database, a report builder, requirements traceability, and
integration with other tools such as defect tracking and configuration
management tools.


A Contender?
------------

The one name that keeps popping up as an alternative to TET is
DejaGnu.  I hear that DejaGnu is used in some college-level classes on
testing methodologies.  It's used as a test harness by the GNU project
and many other organizations as well.  

Some argue that DejaGnu is more suited to testing interactive programs
because it's built on "expect" (see my last column).   But TET can run
expect scripts, and I've seen mention of plans for APIs for tcl and
expect.  TET is more portable than DejaGnu, because expect is firmly
embedded in Unix.  Expect insiders insist that expect needs Unix-style
pseudo terminals, and even systems such as Windows NT that have a
vaguely similar mechanism are not a likely target for an expect port.
It's interesting to note that both TET and expect have mechanisms to
support remote systems that can't run the tool directly.

That's two columns in a row that I've mentioned DejaGnu without giving
any details.  If there's more interest in this subject, please send in
your comments.  For that matter, any comments at all on what you'd
like to see in this column are welcome.


So what's the point?
--------------------

Is TET on its way to becoming the one true test harness?  Well, it's
fighting an uphill battle.  One thing I've discovered when talking with
people about test harnesses is that the choice of a test harness seems
to be a religious issue almost equal to the choice of a programming
language or operating system.  Many companies have made a great deal of
effort developing their own test tools, and they will defend the merits
of their own tools to the death.  I found some people who were
impressed with TET, but many others who don't think it amounts to
much.

A good summation of where we're at with respect to test tools comes
from the Tetworks web page:  "TET is not so much a great technical
achievement as it is a philosophy."  Maybe I'll change my job title to
"alchemist".


For more information
--------------------

The TET Workshop has a new web page at http://tetworks.xopen.org, with
lots of good information about TET.  The old TET page at X/Open's main
web site is also still available at
http://www.xopen.co.uk/public/test/tet.htm.  If you don't have web
access, you can get all three TET implementations via ftp at
ftp.xopen.org.  You can join the tetworks mailing list by sending a
request to postmaster@xopen.org.

Andrew Josey is the contact for TET at X/Open.  You can reach him in
England by email at a.josey@xopen.org, phone +44 1734 508311 ext 2250,
or fax +44 1734 500110.

For more information on PureTestExpert, see Pure's web page at 
http://www.pure.com.  Or call:
    US (408) 720-9200 
    The Netherlands +31 (0)23 569 4300
    Japan +81 3 3863 9285