OrderOfElements - Revision history

JaroslavTulach: /* PS */

JaroslavTulach — Fri, 24 Feb 2012 07:55:30 GMT

←Older revision		Revision as of 07:55, 24 February 2012
Line 73:		Line 73:


-	===== PS =====	+	===== Happy End =====

-	I'll continue ~~to fix~~ the [[NetBeans]] [[JUnit]] tests, but ~~I'll~~ also ~~send~~ a ~~note to [~~[~~JUnit]] gang~~. ~~I am currious to see if there will be an interest to replace~~ the ~~''randomness''~~ of tests ~~with a bit more~~ [https://github.com/KentBeck/junit/pull/293 ~~order] or at least make them [[RandomizedTests\|randomized]~~].	+	I'll continue fixing the [[NetBeans]] [[JUnit]] tests, but to be good open source members, we also donated a [https://github.com/KentBeck/junit/pull/293 patch] that makes the execution order repeatable, but not predicatable (as in unit tests one should not depend on order of tests, right?). The [https://github.com/KentBeck/junit/pull/293 patch] has been accepted in February 2012.

	{{:Talk:OrderOfElements}}		{{:Talk:OrderOfElements}}

	[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]		[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

JaroslavTulach: /* PS */

JaroslavTulach — Sat, 27 Aug 2011 16:23:10 GMT

←Older revision		Revision as of 16:23, 27 August 2011
Line 75:		Line 75:
	===== PS =====		===== PS =====

-	I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more order or at least make them [[RandomizedTests\|randomized]].	+	I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more [https://github.com/KentBeck/junit/pull/293 order] or at least make them [[RandomizedTests\|randomized]].

	{{:Talk:OrderOfElements}}		{{:Talk:OrderOfElements}}

	[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]		[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

JaroslavTulach at 16:22, 27 August 2011

JaroslavTulach — Sat, 27 Aug 2011 16:22:22 GMT

←Older revision		Revision as of 16:22, 27 August 2011
Line 71:		Line 71:

	<comments/>		<comments/>
		+

	===== PS =====		===== PS =====

	I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more order or at least make them [[RandomizedTests\|randomized]].		I'll continue to fix the [[NetBeans]] [[JUnit]] tests, but I'll also send a note to [[JUnit]] gang. I am currious to see if there will be an interest to replace the ''randomness'' of tests with a bit more order or at least make them [[RandomizedTests\|randomized]].
		+
		+	{{:Talk:OrderOfElements}}

	[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]		[[Category:APITypes]] [[Category:APIDesignPatterns:Evolution]] [[Category:APIDesignPatterns:Anti]]

JaroslavTulach at 13:30, 22 August 2011

JaroslavTulach — Mon, 22 Aug 2011 13:30:49 GMT

←Older revision		Revision as of 13:30, 22 August 2011
Line 7:		Line 7:
	Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.		Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

-	However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7 ~~to see it~~:	+	However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''!
		+
		+	=== Random Order of Tests ===
		+
		+	After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!
		+
		+	Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.
		+
		+	I'm not sure about what motivated the change, but if this is not a violation of [[BackwardCompatibility]] with respect to the specification, it is clearly a [[BackwardCompatibility]] problem with respect to runtime and good habits! Just execute following test few times on [[JDK]]6 and [[JDK]]7 and compare the results:

	<source lang="java">		<source lang="java">
Line 29:		Line 37:
	}		}
	</source>		</source>
-
-
-	~~=== Random Order of Tests ===~~
-
-	After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!
-
-	Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.
-
-	I'm not sure about what motivated the change, but if this is not a violation of [[BackwardCompatibility]] with respect to the specification, it is clearly a [[BackwardCompatibility]] problem with respect to runtime and good habits!

	== What can be done about [[JDK]]7 incompatibility? ==		== What can be done about [[JDK]]7 incompatibility? ==

JaroslavTulach: /* JUnit and Switch to JDK7 */

JaroslavTulach — Mon, 22 Aug 2011 13:29:30 GMT

JUnit and Switch to JDK7

←Older revision		Revision as of 13:29, 22 August 2011
Line 7:		Line 7:
	Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.		Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

-	However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7:	+	However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7 to see it:

	<source lang="java">		<source lang="java">

JaroslavTulach: /* Random Order of Tests */

JaroslavTulach — Mon, 22 Aug 2011 13:28:12 GMT

Random Order of Tests

←Older revision		Revision as of 13:28, 22 August 2011
Line 34:		Line 34:

	After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!		After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!
-

	Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.		Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.

JaroslavTulach at 13:27, 22 August 2011

JaroslavTulach — Mon, 22 Aug 2011 13:27:58 GMT

←Older revision		Revision as of 13:27, 22 August 2011
Line 7:		Line 7:
	Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.		Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.

-	However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''!	+	However there is a much bigger problem: basically none of our test suites are able to pass reliably on [[JDK]]7 (while they pass in 99% cases on [[JDK]]6). To make things even worse, the failures are '''random'''! Just execute following test few times on [[JDK]]7:
-		+
-	~~=== Random Order of Tests ===~~	+
-		+
-	~~After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random~~ on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]! Just execute following test few times:	+

	<source lang="java">		<source lang="java">
Line 33:		Line 29:
	}		}
	</source>		</source>
		+
		+
		+	=== Random Order of Tests ===
		+
		+	After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!

JaroslavTulach: /* JUnit and Switch to JDK7 */

JaroslavTulach — Mon, 22 Aug 2011 13:23:15 GMT

JUnit and Switch to JDK7

←Older revision		Revision as of 13:23, 22 August 2011
Line 3:		Line 3:
	== [[JUnit]] and Switch to [[JDK]]7 ==		== [[JUnit]] and Switch to [[JDK]]7 ==

-	[[NetBeans]] uses [[JUnit]] a lot for its own internal testing. For historical reasons, the majority of our existing tests are written using [[JUnit]]3 style and even newly created tests continue to be based on the 3.x style. I want to mention this at the beginning, as I am not sure if the same problems discussed below impact [[JUnit]]4 style tests (although I believe they do).	+	[[NetBeans]] uses [[JUnit]] a lot for its own internal testing. We have invested a lot of time in stabilizing our tests over the last four years. More than 8,000 NetBeans tests are known to pass reliably and about 1,000 others are known to be stable in more than 99% of cases (we mark these as ''randomly failing'' tests).
-		+
-	We have invested a lot of time in stabilizing our tests over the last four years. More than 8,000 NetBeans tests are known to pass reliably and about 1,000 others are known to be stable in more than 99% of cases (we mark these as ''randomly failing'' tests).	+

	Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.		Anyway, with [[JDK]]7 being out, we started to execute the tests on both [[JDK]]s (6 and 7) to see how many regressions we have. Not being naive, we expected that there would be some (for example [[JDK]]7 defines a broken property editor for all enum types which takes precedence over our own property editor used on [[JDK]]6 and tests verifying the correct behavior correctly fail). But that is OK, one cannot improve a framework without making its [[amoeba]] shape shiver a bit.
Line 13:		Line 11:
	=== Random Order of Tests ===		=== Random Order of Tests ===

-	After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]!	+	After a week of investigation, I realized and proved (by reading the log files), that the order of executed '''testXYZ''' methods is random on [[JDK]]7. As the version of [[JUnit]] remains unchanged and as it only calls {{JDK\|java/lang\|Class}}.'''getMethods()''', the only framework to blame for shaking like an [[amoeba]] is the [[JDK]]! Just execute following test few times:
		+
		+	<source lang="java">
		+	package org.bar;
		+
		+	import org.junit.Assert;
		+	import org.junit.Test;
		+
		+	public class OrderedTest {
		+	private static int counter;
		+
		+	@Test public void testZero() {
		+	Assert.assertEquals(0, counter);
		+	counter++;
		+	}
		+
		+	@Test public void testOne() {
		+	Assert.assertEquals(1, counter);
		+	counter++;
		+	}
		+	}
		+	</source>
		+

	Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.		Sure, reading the [[Javadoc]] of the '''getMethods''' method makes it clear that the order of returned methods can be random. But normal [[API]] users are known to be [[clueless]]! Nobody reads [[javadoc]] while things work. And things used to work for the last four years! [[JDK]]6 (and possibly also [[JDK]]5) return the methods in a stable order, defined by the order of methods in the source.

JaroslavTulach: /* Summary */

JaroslavTulach — Mon, 22 Aug 2011 12:27:30 GMT

Summary

←Older revision		Revision as of 12:27, 22 August 2011
Line 49:		Line 49:
	Clearly, the order of elements returned from an [[API]] may be very significant, especially when they depend in some way on each other - like (inproperly isolated) [[JUnit]] tests.		Clearly, the order of elements returned from an [[API]] may be very significant, especially when they depend in some way on each other - like (inproperly isolated) [[JUnit]] tests.

-	Changing an [[API]] that used to return objects in particular order to return them ''randomized'' is huge [[BackwardCompatibility\|incompatible]] change in the [[Chapter 11\|runtime behavior]] of your [[API]].	+	Changing an [[API]] that used to return objects in particular order to return them ''randomized'' is a huge [[BackwardCompatibility\|incompatible]] change in the [[Chapter 11\|runtime behavior]] of your [[API]].

	<comments/>		<comments/>

JaroslavTulach: /* Random vs. RandomizedTests */

JaroslavTulach — Mon, 22 Aug 2011 12:26:16 GMT

Random vs. RandomizedTests

←Older revision		Revision as of 12:26, 22 August 2011
Line 41:		Line 41:
	First of all, assuming reproducibility is of the biggest value, [[JUnit]]3 could sort all the test methods into fixed order (at least when running on [[JDK]]7 and future releases). This might cause one time failures, but since then all the test would remain stable. Fixing the one time failure would be straightforward anyway, as it would be naturally reproducible.		First of all, assuming reproducibility is of the biggest value, [[JUnit]]3 could sort all the test methods into fixed order (at least when running on [[JDK]]7 and future releases). This might cause one time failures, but since then all the test would remain stable. Fixing the one time failure would be straightforward anyway, as it would be naturally reproducible.

-	In case the ''randomness'' is a feature, rather than defect, [[JUnit]] could be enhanced to support reproducibility. For example it could print the order of tests that lead to a failure as part of the failure message or (as advocated at the end of [[RandomizedTests]] overview), it could even generate code to execute the ~~test~~ in proper order.	+	In case the ''randomness'' is a feature, rather than defect, [[JUnit]] could be enhanced to support reproducibility. For example it could print the order of tests that lead to a failure as part of the failure message or (as advocated at the end of [[RandomizedTests]] overview), it could even generate code to execute the tests in proper order.

	Maybe ''randomness'' is essential. Then the [[JUnit]] could even mix the methods before execution to increase the likehood of the failure (I still don't know when [[JDK]]7 mixes the methods, most of the time the order is stable on my local machine). Then it could be enough to print the seed as part of the assert and support some special mode to run with a fixed seed (again as advocated at [[RandomizedTests]]).		Maybe ''randomness'' is essential. Then the [[JUnit]] could even mix the methods before execution to increase the likehood of the failure (I still don't know when [[JDK]]7 mixes the methods, most of the time the order is stable on my local machine). Then it could be enough to print the seed as part of the assert and support some special mode to run with a fixed seed (again as advocated at [[RandomizedTests]]).