OrderOfElements

From APIDesign

Revision as of 10:58, 22 August 2011

Runtime BackwardCompatibility can be significantly influenced by changing the order of elements in a Collection or in an Array. Sure, some people would argue that depending on the order of items as served by HashSet.iterator() is unreasonable (and there is no doubt about that), however there are less obvious cases.

JUnit and Switch to JDK7

NetBeans is using JUnit a lot for own internal testing. For historical reasons majority of our existing tests is written against JUnit3 and even newly created tests continue to be based on the 3.x style. I want to mention this at the beggining, as I am not sure if the same problem we are facing now influences JUnit4 style tests (although I believe it does).

We have invested a lot of time to stabilize our tests during last four years. More than eight thousands tests is know to pass reliably and about one thousand others is known to be stable in more than 99% of cases (we mark these as randomly failing tests).

Anyway, with JDK7 being out, we started to execute the tests on both JDKs (six and seven) to see how much regressions we have. Not being naive, we expected there'll some (for example we found out that JDK7 defines a broken property editor for all enum types which takes precedence over our own property editor used on JDK6). But that is OK, one cannot improve the framework without making its amoeba shape shiver a bit.

However there is much bigger problem: basically none of our test suites is able to pass reliably on JDK7 (while they pass in 99% cases on JDK6). To make things even worse, the failures are random!

Random Order of Tests

After a week of investigation I realized and proved (by reading the log files), that the order of testXYZ is random on JDK7. As the version of JUnit remains unchanged and as it only calls Class.getMethods(), the only framework to blame for shaking like amoeba is the JDK!

Sure, reading the Javadoc of the getMethods method makes it clear that the order of returned methods can be random, but c'mon we are all clueless! Nobody reads javadoc until necessary and things work. And things used to work for last four years! JDK6 (and possibly also JDK5) used to return the methods in stable order defined by order of methods in the source.

I am not sure what motivated the change, but if this is not a violation of BackwardCompatibility with respect to the specification, it is clearly BackwardCompatibility problem with respect to runtime and good habits!

What can be done about the JDK7 incompatibility?

We want our tests to pass on JDK7. NetBeans usually support multiple releases of JDKs and for close future, we are going to continue to compile with JDK6 and we can run our tests with JDK6 primarily. However one day we will drop support for JDK6 and then we need to have stable, not randomly failing tests running on JDK7. What can we do?

Fix JDK7

Obviously, the simplest way to fix the problem is to change the JDK code to behave as it used to in JDK6. However, given the specification explicitly permitting random order of returned methods, I don't believe there is a force on the planet to make JDK team to do such change. Moreover, they may even be legitimate reasons why this change had to be done (performance comes to my mind).

Learn to Write Proper Tests!

I am sure that progenitors of JUnit have been ready to advice me to learn to write proper unit tests since beginning of reading of my text. Yes, they are right. Proper unit tests are not supposed to silently depend on their execution order. We should make them more robust!

How should we have known? On JDK6 the execution order is always the same. Thus there is almost no chance to run into a problem. On JDK7 the order is random, so we may be getting random failures for months before we eliminate them all (some of them caused by sticky ThreadContextClassLoader for example).

Moreover even if there is a failure on the server, when I re-run the test locally, everything works!

Random vs. Randomized Tests

TBD