Open Testware Reviews Blotter 2004

Copyright 2004 by Tejas Software Consulting - All rights reserved.

click here for Blotter entries for 2003

2004-December-31

Just posted a Browser-Based Testing Survey, discussing how to pull Internet Explorer's puppet strings to facilitate web testing.

2004-December-23

The Scripting Language Survey is now freely available on the storefront page, while I'm working on editing a series of commentaries from advocates of several different languages. If you would like to write about your experience with scripting languages, let me know.

2004-November-12

I have been evaluating web link checkers for several weeks. The tools kept finding broken links on my web sites, so I kept getting distracted trying to fix the links rather than reviewing the tools. But I did manage to leave plenty of bad links so the tools would still have something to find.

Because the quality of these tools varies over a broad range, from "works out of the box" to "how did this ever compile?", I decided to write the review in the Technology Bulletin format, touching on a few of the tools that look the most useful. So do a little drum roll and take a look at the Technology Bulletin: Web Link Checkers.

2004-November-2

Here's a note on a real-world use of data comparator tools. This week I helped a client compare two large and almost identical comma separated value (csv) files using the GNU diff utility on Windows. I use the Cygwin port of diff on my Windows system. But it's a pain to do a minimal Cygwin install just to get one utility, so I point my client to the GnuWin32 collection on SourceForge. Here you can choose the individual tools that you need.

My compatriot installs diff from the GnuWin32 page and runs it from a DOS shell (using the full path to the program, since the installation folder was not added to the system path during installation). As I often see on the first try, it shows that every line in the file is not a match. We add the "-w" option to the diff command line to tell it to ignore white space. After finding that all lines are still not matching, we examine closer and notice that one file has double quotes around some items while the other file has no double quotes at all. We load the file into Notepad and use a search and replace to get rid of the quotes in a copy of the file.

Still we have the same voluminous output from diff, with all lines not matching. So we copy the first line from both files into new files and start lopping text off of the end of the lines until we can finally get diff to flag a match. We find our culprit about halfway across - a "0" in a column in one file where there's nothing in that column in the other file. We check the original files and find that the entire column is a mismatch between the two files, with more serious differences further down than the first one we saw. We likely have a bug on our hands. But we run out of steam before we find a reasonable way to find other mismatched columns. We need a better tool.

I send out a call to the swtest-discuss mailing list to try to find a diff-like tool that tells us not only which lines are not a match but also where on the line it doesn't match, or which cells in a spreadsheet file don't match. Luckily I get a quick volley of responses with numerous ideas. Two of them are worth repeating here. First, in this case we really want diff to check columns rather than rows. Ron Parks points out that we can do that by transposing the data in the csv files. We can import the csv files into OpenOffice Spreadsheet or Microsoft Excel, copy all of the data to the clipboard, open a new spreadsheet file, select Edit->Paste Special, then click Transpose and OK. That will turn the data on its ear so when diff reads a line, it's looking at a column from the original file. We save back to csv files and diff then shows only a single line not matching. That tells us that the one column we found before is the only mismatch.

Walter Görlitz then replies to my query to remind me of the KDiff3 tool. This tool, which runs on Unix, Linux, and Windows, give me exactly what I need, and has a GUI interface as well. This tool will show me not only the lines that don't match, but also the characters on the lines that don't match, so I can see which columns in my data are out of whack.

So perhaps I could have just mentioned KDiff3 and made this much shorter, but where's the fun in that? It's good to look at the problem-solving process and have more than one option available. By the way, one of the first comparison programs my client had tried was the DOS fc program. I quickly went to diff instead, though I see now that fc does have an option to collapse white space. However, it can't completely ignore white space, only collapse consecutive white space characters down to one, so there are some cases where white space could still cause false failures.

2004-September-26

Chris Cleveland of Dieselpoint, Inc. has restored the web site for the web load test tool Dieseltest. However, he says that it hasn't been updated since 2001.

2004-September-24
 
Earlier I promised to point out a few additional web load test tools. I've posted reviews of OpenSTA and The Grinder previously. I've found quite a few web load test tools, and I think this is the type of tool that can really thrust free test tools into the forefront, if for no other reason than the extremely high cost of commercial load test tools. But I'm still looking for one that's production quality.

Here are a few others that I haven't reviewed yet and are worth investigating.
If you can provide any opinions about these or other load test tools, please let me know.

2004-September-11

Mirek Wójtowicz, the author of MWSnap, was away from the office when I sent my queries to him. He has returned and indicates that  the project isn't as dormant as it appears:

The MWSnap project is alive, version 4 is already in late Alpha state. I can't give any specific dates yet, but I hope to release MWSnap 4 still this year.

2004-September-3

My review of The Grinder is now available to non-subscribers.

2004-August-31

Posted a review of MWSnap, a screen capture tool for Microsoft Windows.

2004-August-25

Alan Richardson has changed the way I look at spreadsheets as a test tool. His "Test Utilities in Excel" web page discusses a single spreadsheet file that includes several simple test tools. Here's the important point - you can run these tools on any computer that has Microsoft Excel installed, because it's written using Microsoft's VBA (Visual Basic for Applications).

The testingUtilities spreadsheet includes reimplementations of a few of the features in James Bach's perlclip program that I wrote about in the July 2 entry below, including "counterstrings." There's a flexible file locking feature that you can use to test an application's ability to handle its files getting locked by other applications. The application I'm testing didn't react well at all when I tried out this test on its database file. Other features include a front end to the DOS "fc" command for doing simplistic file comparison, and a button that runs a DOS command-line shell.

Alan explains that on different Windows systems he has access to different languages, but practically all of them he runs across has Microsoft Excel installed. So his nifty spreadsheet tool, while using an odd user interface, works just about everywhere he goes. I was so impressed with this notion that I offered to collaborate on an article with Alan about using tools in environments that we can't control. You'll see the result as a column on StickyMinds within a few weeks.

2004-August-2

These features are now publicly available: QMTest review, Data Comparator Survey, Technology Bulletin: System Call Hijacking Tools, and  Screen Capture Tools Survey.

2004-July-31

The Stress Test Tools Survey is now posted. This was my first attempt to carve off a few of the many free tools that I've found and lumped under "Load and Performance Tools" in order to write a survey article.

I'm also working on a brief listing of some of the more complex web load testing tools like The Grinder and OpenSTA that I've reviewed before, to show more examples of tools in that category.

2004-July-2

James Bach recently sent me a script called perlclip that had evolved from a simple one-liner he and I had collaborated on while at this year's Austin Workshop on Test Automation. Back at the workshop, I showed him my often-used snippet of Perl code that can quickly generate large volumes of test data, and he challenged me to enhance it to shove the data directly into the Windows clipboard. A short while later, I showed him this code:
use Win32::Clipboard; $clip=Win32::Clipboard(); $clip->Set("a" x (1024 * 1024));
Assuming you have Perl and the Win32::Clipboard module installed, this will put a million "a" characters in the clipboard, ready to paste into the application that you're testing.

The perlclip script supports this use with a simple command line interface, where you only need to type the Perl code to create the data (in this case, "a" x (1024 * 1024)). James added a few other features that do interesting things without requiring the user to have any Perl knowledge--one to generate a range of 255 different ASCII characters, one to load a file into the clipboard, and one that generates a "counterstring". James described the counterstring feature in a recent blog entry, and it looks like a nice mechanism for testing with strings where you can tell by looking at them exactly how long they are. For example, if I ask for a 15-character counterstring, I get "*3*5*7*9*12*15*", with the final asterisk marking the 15th character.

Perlclip comes with a self-contained Windows executable that runs without having Perl installed. If anyone is interested in getting it to work on other operating systems, let me know.

2004-May-31

A review of the Mantis bug tracking tool is now posted.

2004-May-27

A reader pointed me to an interesting blog entry by Tim Bray on Jython and dynamic languages. Perhaps "dynamic languages" is emerging as a better term than "scripting languages" - you may recall my difficulty in defining what a scripting language is in the Scripting Language Survey. This article and some of the ones it references make an interesting argument in favor of dynamic languages over the static languages that have long been thought to be better for large programming projects. Interestingly enough, the rise of test-driven development is a key factor.

Tim writes off Perl because he doesn't like its syntax. So I had to endure yet another detractor of my favorite language. But then he mentions something that really makes me think about the future. He mentioned the new Java-like scripting language called Groovy, which was also discussed by my local Java users group recently. Groovy, Jython, and a few other dynamic languages run on the Java virtual machine, so they can all use the Java libraries. For a while I've thought it's a shame that the libraries for dynamic languages have to be reimplemented for each language, resulting in a huge duplication of effort for our scarce open source resources. But the Java VM gives us a way to avoid that. And that, I think, will eventually contribute more to the death of Perl than all the whining about syntax. Whether it's the Java VM that wins out will depend on the usefulness of its libraries. Perl 6 will have its own VM, Parrot, but I haven't yet wrapped my brain around the implications.

2004-May-14

Sorry I didn't get a review out last month. I'll have my review of Mantis posted soon. In the meantime, I've opened up three more features for free access: Holodeck Enterprise Edition (Trial Version), Test Design Tool Survey, and JUnit. Also, a quick mention of a web automation tool that came in very handy is below.

2004-May-14

While reviewing the Mantis bug tracking tool, I had the opportunity to explore a powerful web automation library, Perl's WWW::Mechanize, developed by Perl testing aficionado Andy Lester. I wanted to use Mantis with a decent volume of data already in the database, and I thought WWW::Mechanize might provide a quick way to dump lots of bugs into the database via the http interface for Mantis. Within an hour or two, I had my script, and after an overnight run, I had about 16,000 bugs in the database. One of the scripts I used is below. I actually ran two different scripts concurrently, submitting as two different users, and one of them died prematurely with an out of memory error.

This gave me a good test bed to explore performance and scalability issues, and I also filed these two bugs against Mantis: "Locked out of View Bugs page after large query", and "Race condition in summary calculations causes inconsistent totals".

For more information, see the article Screen-scraping with WWW::Mechanize, and for more about automating web interfaces, the book Spidering Hacks looks like it might be a good reference. Installing the library requires installing a long list of other prerequisite libraries, which Perl's CPAN module handled with uncharacteristic grace when I installed under Cygwin and Debian Linux (perl -MCPAN -e 'install WWW::Mechanize').

Here's the Perl script I used to populate the database. Of course, a more sophisticated script could have generated more varied and realistic data.

#!/usr/bin/perl

use warnings;
use strict;
use WWW::Mechanize;
use HTTP::Cookies;

my $agent;

sub pr {
  print $agent->content();
  print "\n--------------------------------\n";
}

$agent = WWW::Mechanize->new();
$agent->cookie_jar(HTTP::Cookies->new);
die if ! $agent->get("http://mantis-server/mantis");
≺
die if ! $agent->submit_form(
  form_name => 'login_form',
  fields => {
    username => 'foo',
    password => 'foo'
  },
);
≺
die if ! $agent->follow_link( text => "Report Bug");
≺
die if ! $agent->submit_form(
  fields => {
    project_id => 1
  }
);
≺
for my $loop (1..10000) {
  print "LOOP $loop\n";
  die if ! $agent->submit_form(
    form_name => 'report_bug_form',
    fields => {
      summary => "this is the summary of the bug I'm submitting",
      description =>
"Wow, is this quite the bug.  Nothing seems to work.
It won't even compile.  Who wrote this stuff anyway?
My five-year-old could write better software.  I could go
on.  In fact, I will.  This is the buggiest bug I've ever
seen.  Boy, it bugs me.

Somebody ought to fix this bug.  Real soon now."
    }
  );
  ≺
  die if ! $agent->follow_link( text => "Report Bug");
  ≺
}

2004-March-24

Things are never as simple as they seem. In our review of jenny, jenny beat ALLPAIRS on test suite optimization on all of the synthetic tests we tried. But today I was helping someone use the pairwise technique and discovered that ALLPAIRS does beat jenny on optimization on half of the real-world problems I set up, and they tied on the rest.

size of each dimension
 ALLPAIRS test count
 jenny test count
5 3 3 2 2
 16
 16
3 2 2 2
 6
 8
3 2 2
 6
 7
3 2
 6
 6

So we have to keep in mind that neither program has a perfectly optimized algorithm. If it's really critical that you squeeze your test suite as small as possible, try using both tools and see which one does best.

To facilitate using jenny without having to handcraft a sed script for each run, I wrote a Perl script that makes jenny's interface largely like ALLPAIRS, minus the diagnostic output. It uses exactly the same input file format as ALLPAIRS. I call it nicejenny. I'm placing it in the public domain, so feel free to share it, but beware it's just the result of an afternoon of hacking.

2004-March-22

Do you test open source software? I am trying to gather a community of people on the swtest-discuss mailing list who would like to discuss open source testing and contrast it with commercial testing practices. If you join in, please send an introduction and describe what you've done to test open source programs.

Last year I recommend the free-testing-tools mailing list for those of you who wanted an interactive discussion forum about freeware test tools. The free-testing-tools list isn't showing any hope to achieving critical mass, and I no longer recommend it. But feel free to use the swtest-discuss list for any general discussions about free test tools. Note that if you have questions about a specific tool, you'll have more success using a mailing list that's specific to that tool.

2004-March-19

Posted a review of the pairwise test design tool jenny. Christopher Meisenzahl was our reviewer this time, the first guest reviewer to brave my monstrous review template. He did a great job.

If you're tracking the content that's released to the public, note that along with the May 2003 blotter entry comes the InstallWatch review and the GUI Test Driver Survey that are now publicly available from the storefront page.

2004-March-19

Bret Pettichord offers additional insights on why tool vendors might not be giving straight answers about the differences between orthogonal arrays and the all-pairs technique.

There is a reason why the experts aren't always clear about such things. The advantage of orthogonal arrays that you allude to here (that it helps zero in on the relevant test parameters) is in fact erroneous. But something kinda like this is kinda sorta true in completely different contexts (namely experiments that require linear regression statistical analysis). But if you are an expert selling an expensive orthogonal array tool, then you don't want to admit that. So you say something confusing that suggests that your tool is better.

2004-February-24

Bret Pettichord commented that this statement in the ALLPAIRS review doesn't make sense: "Using orthogonal arrays allows for some sophisticated failure analysis that you can't do with all-pairs..."

After discussing it with Bret, I realize that I'd have to expand my knowledge of higher math before I can draw any intelligent conclusions about how orthogonal arrays might offer any more value than all-pairs. He pointed to Elisabeth Hendrickson's high-level explanation: "The chief difference between the two test reduction methods is that it is possible for a set of test data to meet the all pairs condition without meeting the orthogonal arrays condition, though orthogonal arrays will give you all pairs."

Stay tuned for an analysis of another all-pairs tool, jenny, written by our first guest reviewer to brave the full in-depth review template.

2004-February-16

Posted a review of FitNesse, a wiki-based acceptance test automation tool.

Newly released to the public: the Sclc Metrics Tool review and the Unit Test Tool Survey. Now available from the publicly-accessible version of the Blotter.

Items I'm currently researching for future content include a bug tracking tool called Mantis, and maybe I'll finally summon the courage to dive into surveying part of the huge field of load testing tools.

2004-February-14

I have started to open up some Open Testware Reviews content to the general public. See the link on the storefront page. I am posting features and blotter entries that are older than nine months old. Subscribers still have exclusive access to newer content.

So you may point your friends to the free content if you've been wanting to share it with them and they're not subscribers yet. Just remember that everything under tejasconsulting.com/open-testware/ is open to the public, and everything under http://tejasconsulting.com/otr/ requires a subscription. When additional features are opened to the public, I'll announce it in the blotter here and also in the messages to Open Testware Announcement subscribers (you can point people to the bottom of the storefront page for subscription information).

2004-February-11

Chris Meisenzahl sent feedback on the ALLPAIRS review:

Just read the ALLPAIRS review again. Very well done. I used ALLPAIRS extensively on a couple of projects last year and liked it quite a bit. I ran into many of the same issues you did but quickly worked around most.

The most limiting part for us was the inability to tell the tool that some combinations were impossible. I worked around this by breaking the input matrix into 2 to 3 different matrices and running ALLPAIRS for each. This left us with some redundancy, but still WAY better than not using the tool.

I gave an example of one way to deal with impossible combinations in the review. James Bach recently pointed out a better method - pre-permute the two problematic attributes, remove the impossible pairs, and then list each remaining combination as a value in a single column of your main matrix. Then run ALLPAIRS as usual. One column will represent two test attributes, and there are no extra tests added to work around the constraint.

I played with this tool a while last summer (just for the 10 day demo). It did permit you to specify impossible combinations: Pro-Test.

I hadn't heard of that one. Thanks. Remember also that the free tool jenny, mentioned in the Test Design Tool Survey, lets you specify constraints.

2004-February-6

Timothy Wall, the author of Abbot, wrote "Java GUI Testing Tools," a comparison of four free GUI test tools for Java. The report covers Abbot, Jemmy, jfcUnit, and Marathon. This is a nice adjunct to the GUI Test Driver Survey I published last May. Timothy gave me permission to give his report a home on the web. It is presented here without editing.

2004-January-30

Chris Meisenzahl and I recently discussed his reaction to the Scripting Language Survey.

Very well done! Is sed worth mentioning? I mention it as it's so often mentioned in the same breath as awk.

If I remember correctly, sed doesn't resemble a general-purpose scripting language like awk does, and it's only useful as a filter. I hear that awk and sed are often used together, however.

The page for the One True Awk is http://cm.bell-labs.com/cm/cs/awkbook/.

Thanks for the reference.

Did you mention that some of these scripting tools allow users to quickly prototype algorithms (e.g. Perl, Ruby, etc.) and GUIs (tcl/tk)?


I didn't, but you just did. In fact, I recently prototyped a GUI test tool using Perl, and then (reluctantly) ported it to C++.

Does Perl actually stand for Practical Extraction and Reporting Language? I was never sure if that was true or not.

Near the top of the perl man page it says "Practical Extraction and Report Language." However, if you read all the way down to the bottom, you'll see where it says "Perl actually stands for Pathologically Eclectic Rubbish Lister, but don't tell anyone I said that." Choose which one you like best.

I'm far from a Perl expert, but there is one thing that bugs me about it and it's kind of the flip side of one of its advantages. Perl has a lot of idioms and syntactic shortcuts. This of course lets people do the same thing in many different manners. In the past this has caused me some confusion when trying to read someone else's code. Just a comment.

Do you think the code you were trying to read was well-written?

Not sure, I wasn't expert enough to say. You bring up a good point though.

It's possible to write unreadable code in any language. Perl just gives people more ways to write bad code. In my Perl tutorial  I like to quote Perl's motto--"There's More Than One Way To Do It (TMTOWTDI)." Then in ominous tones, I recite Python's motto--"There's Only One Way," or the longer and less ominous form: "There should be one--and preferably only one--obvious way to do it." TMTOWTDI is both a blessing and a curse.

2004-January-8

Posted the Scripting Languages Survey, highlighting 10 general-purpose languages and 9 shells, out of the dozens of available options.

2004-January-6

Manfred Rätzmann responded to the discussion about whether reviews should go beyond the out-of-the-box experience -

I'm new to Open Testware Reviews and I like it! My experience with open tools is somewhat frustrating because I can not use them out-of-the-box. Although I have some programming background and wrote a test and automation tool myself, I don't want to spend my time playing with make files and compiler options before using a tool.

So that's a vote to continue to focus on the initial impressions of a tool. Any other opinions?

2004-January-2

Happy 2004! Let's start off with an index to the 2003 features, organized by category. I've also indexed the blotter entries that are relevant to each tool; these are either reader comments, tool author feedback, or updates from me. These are the "followup" notes.

Category
 Content
Defect Tracking Tools Defect Tracking Tools Survey (2003-February)
GUI Test Drivers GUI Test Driver Survey (2003-May, followup 2003-July)
Load and Performance Tools OpenSTA (Open System Testing Architecture) (2003-March, followup 2003-May)

The Grinder (2003-December, followups 2003-December-8, 2003-December-9, 2003-December-26)
Miscellaneous Tools Screen Capture Tools Survey (2003-October)
Test Drivers and Test Suite Management Tools Black Box Test Driver Survey (2003-March)
Static Analysis Tools Sclc Metrics Tool (2003-April, followup 2003-May)
Test Design Tools ALLPAIRS Test Case Generation Tool (2003-February, followups 2003-March, 2003-July)

Test Design Tool Survey (2003-August)
Test Drivers and Test Suite Management Tools QMTest (2003-September)
Test Implementation Tools Data Comparator Survey (2003-September)

Holodeck Enterprise Edition (Trial Version) (2003-August, followup 2003-September)

InstallWatch (2003-May)

Technology Bulletin: System Call Hijacking Tools (2003-October, followup 2003-October-8, 2003-November-19)
Unit Test Tools JUnit (2003-July, followup 2003-August)

Unit Test Tool Survey (2003-April)

Also, of the individual tools I reviewed, four have been updated since the review (it's disappointing that there weren't more new releases). They are:
Stay tuned for a review of FitNesse, a wiki-based test management tool. I'm also toying with the idea of surveying open source scripting languages.

click here for Blotter entries for 2003

  Back to the Open Testware Reviews Subscribers Page