OrderOfElements

From APIDesign

Runtime BackwardCompatibility can be significantly influenced by changing the order of elements in a Collection or in an Array. Sure, some people would argue that depending on the order of items as served by HashSet.iterator() is unreasonable (and there is no doubt about that), however there are less obvious cases.

JUnit and Switch to JDK7

NetBeans is using JUnit a lot for own internal testing. For historical reasons majority of our existing tests is written against JUnit3 and even newly created tests continue to be based on the 3.x style. I want to mention this at the beggining, as I am not sure if the same problem we are facing now influences JUnit4 style tests (although I believe it does).

We have invested a lot of time to stabilize our tests during last four years. More than eight thousands tests is know to pass reliably and about one thousand others is known to be stable in more than 99% of cases (we mark these as randomly failing tests).

Anyway, with JDK7 being out, we started to execute the tests on both JDKs (six and seven) to see how much regressions we have. Not being naive, we expected there'll some (for example we found out that JDK7 defines a broken property editor for all enum types which takes precedence over our own property editor used on JDK6). But that is OK, one cannot improve the framework without making its amoeba shape shiver a bit.

However there is much bigger problem: basically none of our test suites is able to pass reliably on JDK7 (while they pass in 99% cases on JDK6). To make things even worse, the failures are random!

Random Order of Tests

After a week of investigation I realized and proved (by reading the log files), that the order of testXYZ is random on JDK7. As the version of JUnit remains unchanged and as it only calls Class.getMethods(), the only framework to blame for shaking like amoeba is the JDK!

TBD