Hurrah! A stress test tool that I wrote while employed at Convex Computer Corporation has been released with an open source license. It's called "stress_driver" and it's sitting hidden in a 20 meg tar file where no one will likely find it, and anyone who does find it won't know what to do with it. It runs on an operating system version that few people use. It was written using Perl 4, and though it's been ported to Perl 5, it still uses a Perl 4 style, including requiring some header files that are conjured via black magic. But I found that it was a very useful tool, and I bet that it could easily be ported to other operating systems.
I've been talking to Extreme Programming (XP) and other agile development advocates about test-first development. So why not test first maintenance? The idea with test-first development is that when you develop a new feature, you first write a test, you run the test to verify that it fails, you develop the feature, then run the test and see if it passes.
Here I have an 816-line perl script that doesn't run on any system I have access to. There are no tests. I'm going to dive into the deep end and try test-first maintenance for legacy code, while porting stress_driver to the Cygwin environment on Windows NT 4.0. I'm keeping a diary along the way. Here are some highlights and extra commentary.
Oh, by the way, I'm not familiar with Perl's test harness modules, though I know that several exist. Having run the test suite for Perl itself and some of its modules, I choose the same basic "Test" module that they use, and I decide to use Test::Harness in a script that will kick off all of the tests.
2002-05-08
2:08pm
The simplest test I can write is one that uses no command line arguments.
It turns out that this is a negative test - the expected result is an error
message, because at least one argument is required. I don't think agile
developers write a lot of negative tests. Oh well. I write the "badopt"
test. It passes, but I didn't verify the text of the error message. It
turns out that the stress_driver is croaking because I haven't starting
porting it to Cygwin yet. So I add another check based on the error message,
and that fails.
Seems like I have a lot of work to do to get this first test to pass. Hmmm, the XP tenets say I should keep things simple. So I simply comment out the parts that don't work and are preventing the program from getting as far as the code that checks the command line arguments. I have my first passing test!
I decide to add some subtests to "badopt." As a tester, I find negative tests more fun than positive tests. :-) One new test passes a test program path to stress_driver that doesn't exist. (Talking about testing a test tool can be confusing - when using stress_driver, you give it the path of a test program that it will run) Cool, that passes.
2:24pm
I want to add another test that specifies a program that isn't executable.
Now is when I start wishing for a more complete test environment. I need
to create a file and make sure the execute bits are off. Normally, I expect
the test harness to give me a working directory where I can create any
files that are needed. I hack my test harness script so it creates a working
directory and put the path in an environment variable. That subtest passes.
In retrospect, I wonder why I didn't just create a non-executable file
ahead of time in the test suite directory. Maybe because it it's too easy
for file permissions to be botched when installing a test suite.
2:45pm
Okay, I'll force myself to write some positive tests. I create the
second test, named "simple." I'll tell the stress_driver run "sleep 100000"
and then interrupt it shortly after it starts. There is a -life option
that tells stress_driver how long to run. Unfortunately, the lowest it
can go is one minute, which is unacceptable for a test case that should
be able to do its job in a few seconds. I modify stress_driver so that
the -life option can understand seconds as well as minutes.
Testers often have to ask developers to add testability features to their programs. It's such an easier sell when I'm both the tester and the developer. I recall when I originally developed the code, I modified it so that minutes were interpreted as seconds while I was testing, but since I didn't write any reusable tests, I didn't bother to support both.
2002-05-11
9:33am
A big change that I've been planning to make is to rip out my home-grown
event loop and use the Event module instead. I now have 12 subtests in
three files, 15 seconds runtime. All usually pass, but one intermittently
fails in the event code. I decide it's time to do the big changeover rather
than trying to fix the old code.
2002-05-13
2:41pm
All tests are now passing after the event code changeover (and the
code is about 90 lines leaner now). But I'm suspicious - that was too easy.
I examine the logs created from running the "simple" test, and I see that
stress_driver never actually started any test programs. My tests need to
do a lot more verification. I realize that I'm using a unit testing framework
to do high-level functional testing. Verification would be much easier
and more thorough if I were doing true unit testing and had more access
to the program state.
2:49pm
Oops! I learn that when commenting out some of my event code libraries,
I also commented the code that initially starts the child processes, which
is still needed in the new event design. Fixed. I'm glad I tend to comment
out code and test the program before actually deleting the code.
10:05pm
Fixed several other problems, and the post-Event module code now passes
all 12 tests.
2002-05-17
5:19pm
I'm getting tired of setting -life to one second for my positive tests.
It's not elegant, and it's still not as optimized as it could be. So I
give stress_driver a new -iterate option that specifies the maximum number
of times to iterate the test program. For many of my stress_driver tests,
I'll specify just one iteration, and many will complete in less than a
second. I wonder why I never thought to create that feature before. Chalk
up another one for testability.
Well, that's where I'll leave you for now. Along the way, I found bugs in my original stress_driver design (including a minor Y2K bug) as well as the new code I added. I found bugs in the Event module and perl itself, including a reproducible crash in the perl interpreter. I found myself wishing for a more full-featured test environment, so I plan to investigate the other Test modules that are available.
If you're a Perl hacker who's interested in participating in the test-first maintenance project and in using an alpha version of a general-purpose stress test tool, let me know. There's plenty more testing to be done.
Copyright 2002, Danny R. Faught
Danny is a software quality consultant based in Fort Worth, Texas.
He can be reached at faught@tejasconsulting.com
and www.tejasconsulting.com.