Open Testware Reviews

Stress Test Tools Survey

Reviewed: 2004-July-31
Testingfaqs.org category: Load and Performance Tools

About stress test tools

Stress testing and related types of system-level testing are some of the most fun I've had on the job. Perhaps some people prefer to create, but for me, there's nothing better than getting paid for trying to crash something.

The IEEE defines "stress testing" as "Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements" (IEEE Standard Glossary of Software Engineering Terminology). But in practice, the term "stress testing" is applied to any test that causes the software under test to be very busy, whether any defined or undocumented limits are reached or not. "Robustness" is a closely related concept. The IEEE defines robustness as "The degree to which a system or component can function correctly in the presence of invalid input or stressful environmental conditions." Tools that test robustness by sending invalid inputs and checking for proper error handling are often called stress test tools.

The usage of terms describing system-level testing varies wildly. When I scanned my big honking list of free test tools, I found an amorphous blob of tools that variously claim to do stress testing, load testing, performance testing, benchmarking, and profiling, but there's no standardization on what the terms in each project's self-description means. For this survey, I sought out tools that claim to help with stress testing and that don't fall into the category of performance testing. Keep in mind that the other types of system test tools can often also be used as stress test tools.

Stress test tools often have a random element to them. This randomization can be introduced at different conceptual levels:

Dumb monkey - Knows enough about the application to send inputs, but doesn't keep track of the state of the application or what is or isn't valid input. Crashme and fuzz are examples, using purely random input.
Smart monkey - Utilizes knowledge about the operation of the software and may keep track of the current state. Dbgrinder is an example; it uses a profile to select defined actions to execute.

References

Noel Nyman, "Using Monkey Test Tools: How to find bugs cost-effectively through random testing," Software Testing and Quality Engineering, Jan/Feb 2000. The concept is not limited to stress testing.
Danny Faught, "Performance Testing Terms - the Big Picture," Tejas Software Consulting Newsletter, April 2001. I grow less happy with the terms as time goes on, but at least it gives one perspective.

About the matrix

Ten stress test tools are listed here. Some are very focused, some specialize on one or a small number of interfaces and may require a non-trivial amount of configuration before they can be run, and two are generic stress test drivers ("agents," to use a term suggested by Bret Pettichord) that provide the test execution infrastructure but require you to supply the test program.

Tool	Platforms	Notes
Ballista	Unix, Windows (not clearly specified)	Operating system robustness testing tool, the result of a research project led by Phil Koopman at Carnegie Mellon University.
cpuburn	Windows, Linux, BSD	Designed to heavily load CPU chips, Intel P5, & P6, AMD K6 & K7. Stresses the cooling systems, motherboard, and power supply.
crashme	Unix, Windows, OS/2, VMS	A classic robustness test tool that executes random bits of data. Tends to exercise a system's low-level exception handling capabilities, because most of the instructions are invalid.
dbgrinder	Any platform supported by Perl	Induces a randomly generated load on a MySQL database server.
fuzz	Unix, Windows NT	A research project that did random testing on Unix utilities and system calls, network services, and X-Windows and MS Windows GUI events.
Open DTE	Any platform supported by Java	A generic controller for distributing stress tests across multiple machines. GUI interface. The project seems to be dormant.
stress	Unix	A simple tool which imposes a configurable amount of CPU, memory, I/O, and disk stress on a POSIX-compliant operating system.
Stress tool	Any platform supported by Java	Simple web test tool with a command-line interface. Source code not available.
stress_driver	Linux, Windows	Generic randomized stress test scheduler, command-line interface. I designed the tool and later ported it to Linux and Cygwin.
Torture	Unix	Simple web stress test tool, can post randomized data to web forms.