OrderOfElements
From APIDesign
(→JUnit and Switch to JDK7) |
(→JUnit and Switch to JDK7) |
||
| Line 9: | Line 9: | ||
Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (six and seven) to see how much regressions we have. Not being naive, we expected there'll some (for example we found out that [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6). But that is OK, one cannot improve the framework without making its [[amoeba]] shape shiver a bit. | Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (six and seven) to see how much regressions we have. Not being naive, we expected there'll some (for example we found out that [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6). But that is OK, one cannot improve the framework without making its [[amoeba]] shape shiver a bit. | ||
| - | However there is much bigger problem: basically none of our test suites is able to pass reliably on [[JDK]]7 (while they | + | However there is much bigger problem: basically none of our test suites is able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! |
| + | |||
| + | === Random Order of Tests === | ||
| + | |||
| + | After a week of investigation I realized and proved (by reading the log files), that the order of '''testXYZ''' is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK|java/lang|Class}}.'''getMethods()''', the only framework to blame for shaking like [[amoeba]] is the [[JDK]]! | ||
| + | |||
| + | |||
[[TBD]] | [[TBD]] | ||
[[Category:APITypes]] | [[Category:APITypes]] | ||
Revision as of 10:05, 22 August 2011
Runtime BackwardCompatibility can be significantly influenced by changing the order of elements in a Collection or in an Array. Sure, some people would argue that depending on the order of items as served by HashSet.iterator() is unreasonable (and there is no doubt about that), however there are less obvious cases.
JUnit and Switch to JDK7
NetBeans is using JUnit a lot for own internal testing. For historical reasons majority of our existing tests is written against JUnit3 and even newly created tests continue to be based on the 3.x style. I want to mention this at the beggining, as I am not sure if the same problem we are facing now influences JUnit4 style tests (although I believe it does).
We have invested a lot of time to stabilize our tests during last four years. More than eight thousands tests is know to pass reliably and about one thousand others is known to be stable in more than 99% of cases (we mark these as randomly failing tests).
Anyway, with JDK7 being out, we started to execute the tests on both JDKs (six and seven) to see how much regressions we have. Not being naive, we expected there'll some (for example we found out that JDK7 defines a broken property editor for all enum types which takes precedence over our own property editor used on JDK6). But that is OK, one cannot improve the framework without making its amoeba shape shiver a bit.
However there is much bigger problem: basically none of our test suites is able to pass reliably on JDK7 (while they pass in 99% cases on JDK6). To make things even worse, the failures are random!
Random Order of Tests
After a week of investigation I realized and proved (by reading the log files), that the order of testXYZ is random on JDK7. As the version of JUnit remains unchanged and as it only calls Class.getMethods(), the only framework to blame for shaking like amoeba is the JDK!
Follow
