Open Testware Reviews

JUnit

Overview -- Maturity -- Project activity -- Platforms -- Support -- Documentation -- Installation -- Implementation -- Performance -- Similar tools -- Limitations -- Observations -- Appendix: Sample JUnit test

Overview

Reviewed: 2003-June-30
Version reviewed: 3.8.1, 2002-Sept-03
Maintainer: junit.org
URL: http://www.junit.org/
Testingfaqs.org category: Unit Test Tools
License: Common Public License
User interface: GUI, command line

JUnit is probably the most widely used unit test tool on the planet. If you have Java code to test, and developers willing to take responsibility for the quality of their own code, then you should consider jumping on the bandwagon. The only things that make me nervous about JUnit is the lack of a complete tutorial, and a strange lapse in the release cycle over the last several months. Read on for details.

Here's a screen shot of the GUI interface for JUnit, after running the the test suite from the Appendix. The red bar is showing. The goal for the developer is to achieve the celebrated green bar shortly after it goes red.

JUnit screen shot

Maturity

5 - Production (on a scale of 1-5)

The JUnit team has classified the tool on SourceForge as production quality, and from what I've seen so far, I agree. Its broad usage base increases my confidence that the code has been exercised in a wide variety of scenarios.

Project activity

4 - Active (on a scale of 1-5)

There were six releases of JUnit between January 2000 and September 2002, but this pace stops there. There hasn't been another release since September 2002, and a quick scan of the CVS archives show no code changes checked in during the interim. But because the current release is less than a year old, and the junit mailing list continues to be very active, JUnit earns an "Active" rating.

Platforms

Java tools rarely document their supported platforms. SourceForge says the JUnit project is "OS Independent," which seems to be true to a large extent, because Java runs on a wide variety of platforms. It's fairly safe to assume that JUnit will run anywhere that Java runs.

Support

The mailing list for JUnit users is the JUnit list on Yahoo Groups. An alternative outlet is the JUnit Forum on jguru.com. There are also the JUnit announcements and JUnit developer lists hosted on SourceForge, but both are plagued by spam and little else, so they're of no value. The tool's maintainers did not reply to email I sent them directly, perhaps because the wide popularity of the tool generates too many messages for them to handle personally.

SourceForge hosts databases for Bugs, Support Requests, Patches, and Feature Requests. Personally, I find it confusing when all of these databases are used on a project, but it's good to know that there's a formal mechanism to track bug reports.

The JUnit project uses the CVS server on SourceForge for revision control. The README file gives a nice set of release notes for recent minor releases, but there's practically no documentation on the 3.8.1 bug fix release.

I couldn't find any information about commercial support for JUnit, though there are probably hordes of Java contractors who are familiar with JUnit and could provide help. There is a JUnit training course listed at junit.org.

Documentation

There are several places you can turn to for information about JUnit. The main web page is junit.org. Several links there lead to the SourceForge page, junit.sourceforge.net. Those who are familiar with SourceForge would have an easier time navigating the services hosted by SourceForge by going directly to the summary page, sourceforge.net/projects/junit.

There are several documents provided for JUnit:

"Test Infected" is the seminal JUnit article that's often considered required reading for Java programmers. I found as a whole, though, that the available JUnit documentation is not very cohesive, especially for a beginning Java programmer like me. I dug through document after document that mentioned and illustrated JUnit's GUI, but only in the "Frequently asked questions" file did I find the incantation required to actually start the GUI.

There is a full set of JavaDocs, which has comprehensive low-level coverage of the JUnit classes with documentation extracted from the source code. Also available is a long list of articles about various aspects of JUnit and its extensions.

Another source of documentation is the book Test-Driven Development: By Example by Kent Beck, though it's more of a discussion about the development process than a tutorial introduction to JUnit.

Installation

JUnit installs from a 432 kilobyte zip file that expands to just over a megabyte. The only other installation required is adding the path to the junit.jar file to the Java classpath, plus the path to where your test classes will be located. I was very frustrated with trying to get the classpath right and figuring out the package names relative to the classpath, but that's because I'm a Java newbie.

Implementation

JUnit source code is available in a src.jar file in the binary distribution, which can be extracted using your favorite zip utility, assuming you can convince it that it's okay to read a .jar file. JUnit consists of 47 Java files for a total of 3673 non-comment lines of code, plus 986 comment lines.

JUnit is able to test itself, always a notable feature of a test tool. JUnit comes with 91 test cases implemented in 35 Java files, adding up to 1184 non-comment source lines and 131 comment lines. And yes, all tests pass on both platforms I tried JUnit on, finishing in well under 10 seconds.

Performance

There's not much to report concerning performance. JUnit is basically bound to the performance of the Java run-time environment.

If you compare unit testing to the typical black-box test framework, you're likely to notice a dramatic difference in performance. Simple and well-written black-box tests are likely to run for several seconds each. But in the unit test performance mentioned in the previous section, I was able to run more than 10 unit tests per second. This is because unit tests focus on the code with pinpoint precision. There's no need to initialize the entire application in order to run one test.

Similar tools

JUnit is one of the first of a large class of unit test frameworks collectively called "xUnit." Dozens of unit test tools for other languages are patterned after JUnit. By the way, legend has it that JUnit was based on SUnit, a unit test framework for Smalltalk.

In the April 2003 Unit Test Tool Survey, there is one other Java unit test tool listed, GrandTestAuto, that doesn't have nearly the following that JUnit does. There are a handful of tools that deal with mock objects in Java, plus a few others related to web testing that deal with Java in one way or another. There are also many JUnit extensions available.

Though most unit test frameworks are freeware, there are a few commercial Java test tools listed on the testingfaqs.org Unit Test Tools list, AQtest and Test Mentor - Java Edition, and several more will be added soon.

Limitations

At the time I started the review, there were 47 open bugs in the bug database, and 48 open feature requests. There hasn't been any appreciable progress on the bug backlog since the past JUnit release.

Earlier I mentioned that the documentation can be frustrating for beginners.

A few people have expressed concerns about the fact that JUnit stops after the first failure. Actually, JUnit will continue running test cases if a test case fails, as you can tell by the screen shot above. But it won't continue a test case after the first assertion that fails within that test case. The FAQ discusses why this is okay, and explains that if developers keep their test cases small, then this won't cause a problem.

However, having small tests means that you could end up end a large number of test methods, all hand-coded. This doesn't lend itself well to data-driven testing, or other scenarios where many tests are produced by a small amount of test code that loops. If one JUnit test case is a wrapper on top of 1000 tests from a test database, then stopping at the first sub-test failure is a large limitation. (This concern is certainly not unique to JUnit.) A possible workaround is for the test generation process to spit out fully formed Java test methods as a step before you compile the test.

Observations

JUnit is a wildly popular unit testing framework, used primarily by developers. It is only relevant for Java code, unless you link object code from another language into a Java program, or run your test in an external process. Most of JUnit's functionality is accessed only from a Java API, but there is also a command line and a graphical interface for running tests. JUnit is a fairly simple toolset; what's powerful is that it has facilitated a surge in the practice of unit testing.

JUnit is a good choice for a unit test tool if you have Java code and developers who are willing to write unit tests. Ideally, the developers are doing "test-driven development," where the unit testing is an integral part of the development process. If you're developing in more than one language, maintaining a unit test framework for each one will take extra work. I haven't thoroughly investigated which unit test frameworks or integrated development environments support multiple languages, but it's likely that there are commercial tools that can do this.

A selling point for JUnit is that fact that the unit tests are developed in the same language as the product code, so developers can work in an environment that they're comfortable in. This selling point might not be as strong if you're wanting an independent test team to develop black-box tests. If the testers are already familiar with a different automation language, and they don't need white-box access to the product implementation, then they would likely be more productive with a tool designed for black-box testing.

It's exciting to see developers willingly doing unit testing, and doing it well. JUnit can be a valuable toolset to facilitate the unit testing process.

Appendix: Sample JUnit test

The code below implements the minimal structure for a JUnit test case. There are several subtest methods that illustrate various types of assertions. Every method with a name starting with "test" is wrapped up into a suite through the magic of the TestSuite constructor in the suite method. The user interface is provided by the GUI or the command-line test runner, as in "java junit.swingui.TestRunner MyJUnitDemo" or "java junit.textui.TestRunner MyJUnitDemo," so we don't even necessarily need a main() method.

import junit.framework.*;

public class MyJUnitDemo extends TestCase {
  public void testAssertTrue1() {
    assertTrue(1 == 0);
    fail("we never reach this assertion");
  }
  public void testAssertTrue2() {
    assertTrue("truth", 1 == 0);
  }
  public void testAssertEquals1() {
    assertEquals("integer equality", 1, 0);
  }
  public void testAssertEquals2() {
    assertEquals("string equality", "one", "zero");
  }
  public void testAssertEquals3() {
    assertEquals("double equality", (double) 1, (double) .5, .2);
  }
  public void testFail() {
    fail("do-it-yourself");
  }
  public static Test suite() {
    return new TestSuite(MyJUnitDemo.class);
  }
}