===== PS =====

===== Happy End =====

I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more [https://github.com/KentBeck/junit/pull/293 order] or at least make them [[RandomizedTests|randomized]].

I'll continue fixing the [[NetBeans]] [[JUnit]] tests, but to be good open source members, we also donated a [https://github.com/KentBeck/junit/pull/293 patch] that makes the execution order repeatable, but not predicatable (as in unit tests one should not depend on order of tests, right?). The [https://github.com/KentBeck/junit/pull/293 patch] has been accepted in February 2012.

{{:Talk:OrderOfElements}}

{{:Talk:OrderOfElements}}

[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

===== PS =====

===== PS =====

I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more order or at least make them [[RandomizedTests|randomized]].

I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more [https://github.com/KentBeck/junit/pull/293 order] or at least make them [[RandomizedTests|randomized]].

{{:Talk:OrderOfElements}}

{{:Talk:OrderOfElements}}

[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

<comments/>

<comments/>

===== PS =====

===== PS =====

I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more order or at least make them [[RandomizedTests|randomized]].

I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more order or at least make them [[RandomizedTests|randomized]].

[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7 to see it:

However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''!

 

 

 

 

I'm not sure about what motivated the change, but if this is not a violation of [[BackwardCompatibility]] with respect to the specification, it is clearly a [[BackwardCompatibility]] problem with respect to runtime and good habits! Just execute following test few times on [[JDK]]6 and [[JDK]]7 and compare the results:

<source lang="java">

<source lang="java">

}

}

</source>

</source>

== What can be done about [[JDK]]7 incompatibility? ==

== What can be done about [[JDK]]7 incompatibility? ==

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7:

However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7 to see it:

<source lang="java">

<source lang="java">

After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK|java/lang|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!

After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK|java/lang|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!

Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.

Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''!

However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7:

 

 

After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK|java/lang|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]! Just execute following test few times:

<source lang="java">

<source lang="java">

}

}

</source>

</source>

== [[JUnit]] and Switch to [[JDK]]7 ==

== [[JUnit]] and Switch to [[JDK]]7 ==

[[NetBeans]] uses [[JUnit]] a lot for its own internal testing. For historical reasons, the majority of our existing tests are written using [[JUnit]]3 style and even newly created tests continue to be based on the 3.x style. I want to mention this at the beginning, as I am not sure if the same problems discussed below impact [[JUnit]]4 style tests (although I believe they do).

[[NetBeans]] uses [[JUnit]] a lot for its own internal testing. We have invested a lot of time in stabilizing our tests over the last four years. More than 8,000 NetBeans tests are known to pass reliably and about 1,000 others are known to be stable in more than 99% of cases (we mark these as ''randomly failing'' tests).

 

We have invested a lot of time in stabilizing our tests over the last four years. More than 8,000 NetBeans tests are known to pass reliably and about 1,000 others are known to be stable in more than 99% of cases (we mark these as ''randomly failing'' tests).

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

=== Random Order of Tests ===

=== Random Order of Tests ===

After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK|java/lang|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!

After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK|java/lang|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]! Just execute following test few times:

 

 

 

 

 

Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.

Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.

Clearly, the order of elements returned from an [[API]] may be very significant, especially when they depend in some way on each other - like (inproperly isolated) [[JUnit]] tests.

Clearly, the order of elements returned from an [[API]] may be very significant, especially when they depend in some way on each other - like (inproperly isolated) [[JUnit]] tests.

Changing an [[API]] that used to return objects in particular order to return them ''randomized'' is huge [[BackwardCompatibility|incompatible]] change in the [[Chapter 11|runtime behavior]] of your [[API]].

Changing an [[API]] that used to return objects in particular order to return them ''randomized'' is a huge [[BackwardCompatibility|incompatible]] change in the [[Chapter 11|runtime behavior]] of your [[API]].

<comments/>

<comments/>

First of all, assuming reproducibility is of the biggest value, [[JUnit]]3 could sort all the test methods into fixed order (at least when running on [[JDK]]7 and future releases). This might cause one time failures, but since then all the test would remain stable. Fixing the one time failure would be straightforward anyway, as it would be naturally reproducible.

First of all, assuming reproducibility is of the biggest value, [[JUnit]]3 could sort all the test methods into fixed order (at least when running on [[JDK]]7 and future releases). This might cause one time failures, but since then all the test would remain stable. Fixing the one time failure would be straightforward anyway, as it would be naturally reproducible.

In case the ''randomness'' is a feature, rather than defect, [[JUnit]] could be enhanced to support reproducibility. For example it could print the order of tests that lead to a failure as part of the failure message or (as advocated at the end of [[RandomizedTests]] overview), it could even generate code to execute the test in proper order.

In case the ''randomness'' is a feature, rather than defect, [[JUnit]] could be enhanced to support reproducibility. For example it could print the order of tests that lead to a failure as part of the failure message or (as advocated at the end of [[RandomizedTests]] overview), it could even generate code to execute the tests in proper order.

Maybe ''randomness'' is essential. Then the [[JUnit]] could even mix the methods before execution to increase the likehood of the failure (I still don't know when [[JDK]]7 mixes the methods, most of the time the order is stable on my local machine). Then it could be enough to print the seed as part of the assert and support some special mode to run with a fixed seed (again as advocated at [[RandomizedTests]]).

Maybe ''randomness'' is essential. Then the [[JUnit]] could even mix the methods before execution to increase the likehood of the failure (I still don't know when [[JDK]]7 mixes the methods, most of the time the order is stable on my local machine). Then it could be enough to print the seed as part of the assert and support some special mode to run with a fixed seed (again as advocated at [[RandomizedTests]]).