Alex Ruiz's Blog

JUnit 4.8 introduced Categories: a mechanism to label and group tests, giving developers the option to include or exclude groups (or categories.) This post presents a brief overview of JUnit categories and some unexpected behavior I have found while using them.

1. Quick Introduction

The following example shows how to use categories: (adapted from JUnit’s release notes)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

public interface FastTests { /* category marker */ }
public interface SlowTests { /* category marker */ }
 
public class A {
    @Category(SlowTests.class)
    @Test public void a() {}
}
 
@Category(FastTests.class})
public class B {
    @Test public void b() {}
}
 
@RunWith(Categories.class)
@IncludeCategory(SlowTests.class)
@ExcludeCategory(FastTests.class)
@SuiteClasses({ A.class, B.class })
public class SlowTestSuite {}

Lines 1, 2: we define two categories, FastTests and SlowTests. JUnit categories can be defined as classes or interfaces. Since a category acts like a label or marker, my intuition tells me to use interfaces.

Line 5: we use the annotation @org.junit.experimental.categories.Category to label test classes and test methods with one or more categories.

Lines 6, 9: test methods and test classes can be marked as belonging to one or more categories of tests. Labeling a test class with a category automatically includes all its test methods in such category.

Lines 14 to 18: currently programmatic test suites (line 17) are the only way to specify which test categories (line 14) should be included (line 15) or excluded (line 16) when the suite is executed. I find this approach (especially the way test classes need to included in the suite) too verbose and not so flexible. Hopefully Ant, Maven and IDEs will provide support for categories (with a simpler configuration) in the very near future.

Note: I recently discovered ClasspathSuite, a project that simplifies the creation of programmatic JUnit test suites. For example, we can specify we want to include in a test suite all tests whose names end with “UnitTest.”

2. Category Subtyping

Categories also support subtyping. Let’s say we have the category IntegrationTests that extends SlowTests:

public interface IntegrationTests extends SlowTests {}

Any test class or test method labeled with the category IntegrationTests is also part of the category SlowTests. To be honest, I don’t know how handy category subtyping could be. I’ll need to experiment with it more to have an opinion.

3. Categories and Test Inheritance

3a. Method-level Categories

JUnit behaves as expected when test inheritance is combined with method-level categories. For example:

public class D {
  @Category(GuiTest.class)
  @Test public void d() {}
}

public class E extends D {
  @Category(GuiTest.class)
  @Test public void e() {}
}

@RunWith(Categories.class)
@IncludeCategory(GuiTest.class)
@SuiteClasses(E.class)
public class TestSuite {}

As I expected, when running TestSuite, test methods d and e are executed (both methods belong to the GuiTest category and E inherits method d from superclass D.) Nice!

3b. Class-level Categories

On the other hand, unless I’m missing something, I think I found some strange behavior in JUnit in this scenario. Consider the following classes:

@Category(GuiTest.class)
public class A {
  @Test public void a() {}
}

public class B extends A {
  @Test public void b() {}
}

@RunWith(Categories.class)
@IncludeCategory(GuiTest.class)
@SuiteClasses(B.class)
public class TestSuite {}

As we can see, TestSuite should execute the tests in B that belong to the category GuiTest. I was expecting TestSuite to execute test method a, even though B is not marked as a GuiTest. Here is my reasoning:

1. test method a belongs to the category GuiTest because test class A is labeled with such category
2. test class B is an A and it inherits test method a

Therefore, TestSuite should execute test method a. But it doesn’t! Here is a screenshot of the results I get (click to see full size.)

There are two ways to fix this issue, depending on what test methods we want to actually run:

1. Label class B with GuiTest. In this case, both methods, a and b, will be executed.
2. Label method a with GuiTest. In this case, only method a will be executed.

(I’ll be posting a question regarding this issue in the JUnit mailing list shortly.)

4. Categories vs. TestNG Groups

(You saw this one coming, didn’t you?) Categories (or groups) have been part of TestNG for long time. Unlike JUnit’s, TestNG’s groups are defined as simple strings, not as classes or interfaces.

As a static typing lover, I was pretty happy with JUnit categories. By using an IDE, we could safely rename a category or look for usages of a category within a project. Even though my observation was correct, I was missing one important point: all this works great as long as your test suite is written in Java.

In the real world, I’d like to define a test suite in either Ant or Maven (or Gradle, or Rake.) In this scenario, having categories as Java types does not bring any benefit. In fact, I suspect it would be very verbose and error-prone to specify the fully-qualified name of a category in a build script. Renaming a category now would be limited to a text-based “search and replace.” Ant and Maven really need to provide a way to specify JUnit categories, clever enough to be fool-proof.

As you may expect, I prefer the simplicity and pragmatism of TestNG’s groups.

Update #1: my good friend (and creator of the TestNG framework,) Cédric, reminded me that we can use regular expressions to include or exclude groups in a test suite (details here.) This is really powerful!

5. My Usage of Categories

I’m not using JUnit categories in my test suites yet. I started to look into JUnit categories because I wasn’t completely happy with the way we recognized GUI tests in FEST. We recognize test methods or test classes as “GUI tests” if they have been annotated with the @GUITest (provided by FEST.) When a “GUI test” fails, FEST automatically takes a screenshot of the desktop and includes it in the JUnit or TestNG HTML report. The problem is, our @GUITest annotation is duplicating the functionality of JUnit categories.

To solve this issue, I created a JUnit extension that recognizes test methods or test classes as “GUI tests” if they belong to the GuiTest category. At this moment GuiTest is an interface provided by FEST, but I’m thinking about letting users specify their own GuiTest category as well.

I also refactored this functionality out of the Swing testing module, expecting to reuse it once I implement a JavaFX testing module :)

You can find the FEST code that deals with JUnit categories at github.

6. Conclusion

Having the ability to label and group tests via categories is really a great feature. I still have some reservations about the practicality of defining categories as Java types, the lack of support for this feature from Ant and Maven (not JUnit’s fault,) and the unexpected behavior I noticed when combining class-level categories and test inheritance.

On the brighter side, categories are still an experimental, non-final feature. I’m sure will see many improvements in future JUnit releases :)

Feedback is always welcome.

Update #2: The nice folks at DZone have posted this entry at Javalobby.