OrderOfElements
From APIDesign
Runtime BackwardCompatibility can be significantly influenced by changing the order of elements in a Collection or in an Array. Sure, some people would argue that depending on the order of items as served by HashSet.iterator() is unreasonable (and there is no doubt about that), however there are less obvious cases.
Contents |
JUnit and Switch to JDK7
NetBeans is using JUnit a lot for own internal testing. For historical reasons the majority of our existing tests is written using JUnit3 style and even newly created tests continue to be based on the 3.x style. I want to mention this at the beggining, as I am not sure if the same problem influencing JUnit4 style tests (although I believe it does).
We have invested a lot of time to stabilize our tests during last four years. More than eight thousands tests is known to pass reliably and about one thousand others is known to be stable in more than 99% of cases (we mark these as randomly failing tests).
Anyway, with JDK7 being out, we started to execute the tests on both JDKs (six and seven) to see how much regressions we have. Not being naive, we expected there would be some (for example JDK7 defines a broken property editor for all enum types which takes precedence over our own property editor used on JDK6 and tests verifying the correct behavior rightfully fail). But that is OK, one cannot improve a framework without making its amoeba shape shiver a bit.
However there is much bigger problem: basically none of our test suites is able to pass reliably on JDK7 (while they pass in 99% cases on JDK6). To make things even worse, the failures are random!
Random Order of Tests
After a week of investigation I realized and proved (by reading the log files), that the order of executed testXYZ methods is random on JDK7. As the version of JUnit remains unchanged and as it only calls Class.getMethods(), the only framework to blame for shaking like amoeba is the JDK!
Sure, reading the Javadoc of the getMethods method makes it clear that the order of returned methods can be random. But common: API users are known to be clueless! Nobody reads javadoc while things work. And things used to work for last four years! JDK6 (and possibly also JDK5) used to return the methods in stable order defined by order of methods in the source.
I am not sure what motivated the change, but if this is not a violation of BackwardCompatibility with respect to the specification, it is clearly BackwardCompatibility problem with respect to runtime and good habits!
What can be done about the JDK7 incompatibility?
We want our tests to pass on JDK7. NetBeans usually supports multiple releases of JDKs and for close future, we are going to continue to compile with JDK6 and we can run our tests with JDK6 primarily. However one day we will drop support for JDK6 and then we need to have stable, not randomly failing tests running on JDK7. What can we do?
Fix JDK7
Obviously, the simplest way to fix the problem is to change the JDK code to behave as it used to in JDK6. However, given the specification explicitly permitting random order of returned methods, I don't believe there is a force on the planet to make JDK team to do such change. Moreover, there may even be legitimate reasons why this change had to be done (performance comes to my mind).
Learn to Write Proper Tests!
I am sure that progenitors of JUnit have been ready to advice me to learn to write proper unit tests since beginning of reading of my text. Yes, they are right. Proper unit tests are not supposed to silently depend on their execution order. We should make them more robust!
How should we have known? On JDK6 the execution order is always the same. Thus there is almost no chance to run into a problem. On JDK7 the order is random, so we may be getting random failures for months before we eliminate them all.
Moreover even if there is a failure on the server, when I re-run the test locally, everything works!
Random vs. RandomizedTests
Usage of a RandomizedTest is fully acceptable. However, random does not mean randomized! The most important property of RandomizedTests is reproducibility. As soon as the execution fails, we should have a way to reproduce the failure. The JUnit+JDK7 combination is quite deadly: it provides randomness, but does not help with easy reproducibility. Could something be done about that?
First of all, assuming reproducibility is of the biggest value, JUnit3 could sort all the test methods into fixed order (at least when running on JDK7 and future releases). This might cause one time failures, but since then all the test would remain stable. Fixing the one time failure would be straightforward anyway, as it would be naturally reproducible.
In case the randomness is a feature, rather than defect, JUnit could be enhanced to support reproducibility. For example it could print the order of tests that lead to a failure as part of the failure message or (as advocated at the end of RandomizedTests overview), it could even generate code to execute the test in proper order.
Maybe randomness is essential. Then the JUnit could even mix the methods before execution to increase the likehood of the failure (I still don't know when JDK7 mixes the methods, most of the time the order is stable on my local machine). Then it could be enough to print the seed as part of the assert and support some special mode to run with a fixed seed (again as advocated at RandomizedTests).
Summary
Clearly, the order of elements returned from an API may be very significant, especially when they depend in some way on each other - like (inproperly isolated) JUnit tests.
Changing an API that used to return objects in particular order to return them randomized is huge incompatible change in the runtime behavior of your API.
<comments/>
PS
I'll continue to fix the NetBeans JUnit tests, but I'll also send a note to JUnit gang. I am currious to see if there will be an interest to replace the randomness of tests with a bit more order or at least make them randomized.
Follow
