OrderOfElements

From APIDesign

Runtime BackwardCompatibility can be significantly influenced by changing the order of elements in a Collection or in an Array. Sure, some people would argue that depending on the order of items as served by HashSet.iterator() is unreasonable (and there is no doubt about that), however there are less obvious cases where this dependence is relevant.

JUnit and Switch to JDK7

NetBeans uses JUnit a lot for its own internal testing. We have invested a lot of time in stabilizing our tests over the last four years. More than 8,000 NetBeans tests are known to pass reliably and about 1,000 others are known to be stable in more than 99% of cases (we mark these as randomly failing tests).

Anyway, with JDK7 being out, we started to execute the tests on both JDKs (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example JDK7 defines a broken property editor for all enum types which takes precedence over our own property editor used on JDK6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its amoeba shape shiver a bit.

However there is a much bigger problem: basically none of our test suites are able to pass reliably on JDK7 (while they pass in 99% cases on JDK6). To make things even worse, the failures are random!

Random Order of Tests

After a week of investigation, I realized and proved (by reading the log files), that the order of executed testXYZ methods is random on JDK7. As the version of JUnit remains unchanged and as it only calls Class.getMethods(), the only framework to blame for shaking like an amoeba is the JDK!

Sure, reading the Javadoc of the getMethods method makes it clear that the order of returned methods can be random. But normal API users are known to be clueless! Nobody reads javadoc while things work. And things used to work for the last four years! JDK6 (and possibly also JDK5) return the methods in a stable order, defined by the order of methods in the source.

I'm not sure about what motivated the change, but if this is not a violation of BackwardCompatibility with respect to the specification, it is clearly a BackwardCompatibility problem with respect to runtime and good habits! Just execute following test few times on JDK6 and JDK7 and compare the results:

package org.bar;
 
import org.junit.Assert;
import org.junit.Test;
 
public class OrderedTest {
    private static int counter;
 
    @Test public void testZero() {
        Assert.assertEquals(0, counter);
        counter++;
    }
 
    @Test public void testOne() {
        Assert.assertEquals(1, counter);
        counter++;
    }
}

What can be done about JDK7 incompatibility?

We want our tests to pass on JDK7. NetBeans usually supports multiple releases of JDKs and for close future, we are going to continue to compile with JDK6 and we can run our tests with JDK6 primarily. However one day we will drop support for JDK6 and then we need to have stable, not randomly failing tests running on JDK7. What can we do?

Fix JDK7

Obviously, the simplest way to fix the problem is to change the JDK code to behave as it used to in JDK6. However, given the specification explicitly permitting random order of returned methods, I don't believe there is a force on the planet to make JDK team to do such change. Moreover, there may even be legitimate reasons why this change had to be done (performance comes to my mind).

Learn to Write Proper Tests!

I am sure that progenitors of JUnit have been ready to advice me to learn to write proper unit tests since beginning of reading of my text. Yes, they are right. Proper unit tests are not supposed to silently depend on their execution order. We should make them more robust!

How should we have known? On JDK6 the execution order is always the same. Thus there is almost no chance to run into a problem. On JDK7 the order is random, so we may be getting random failures for months before we eliminate them all.

Moreover even if there is a failure on the server, when I re-run the test locally, everything works!

Random vs. RandomizedTests

Usage of a RandomizedTest is fully acceptable. However, random does not mean randomized! The most important property of RandomizedTests is reproducibility. As soon as the execution fails, we should have a way to reproduce the failure. The JUnit+JDK7 combination is quite deadly: it provides randomness, but does not help with easy reproducibility. Could something be done about that?

First of all, assuming reproducibility is of the biggest value, JUnit3 could sort all the test methods into fixed order (at least when running on JDK7 and future releases). This might cause one time failures, but since then all the test would remain stable. Fixing the one time failure would be straightforward anyway, as it would be naturally reproducible.

In case the randomness is a feature, rather than defect, JUnit could be enhanced to support reproducibility. For example it could print the order of tests that lead to a failure as part of the failure message or (as advocated at the end of RandomizedTests overview), it could even generate code to execute the tests in proper order.

Maybe randomness is essential. Then the JUnit could even mix the methods before execution to increase the likehood of the failure (I still don't know when JDK7 mixes the methods, most of the time the order is stable on my local machine). Then it could be enough to print the seed as part of the assert and support some special mode to run with a fixed seed (again as advocated at RandomizedTests).

Summary

Clearly, the order of elements returned from an API may be very significant, especially when they depend in some way on each other - like (inproperly isolated) JUnit tests.

Changing an API that used to return objects in particular order to return them randomized is a huge incompatible change in the runtime behavior of your API.

<comments/>

PS

I'll continue to fix the NetBeans JUnit tests, but I'll also send a note to JUnit gang. I am currious to see if there will be an interest to replace the randomness of tests with a bit more order or at least make them randomized.

mbien said ...

Hello Michael, I've been told that TestNG supports ordering of tests. Kent replied to me that methods in JUnit test class should rather be independent. As far as the RFE for JUnit goes - right, Jesse created one. Actually, what Jesse suggest is to sort test methods alphabetically.

--JaroslavTulach 16:14, 27 August 2011 (UTC)