CodeCoverage - Revision history

JaroslavTulach at 05:53, 26 March 2022

2022-03-26T05:53:57Z

←Older revision		Revision as of 05:53, 26 March 2022
Line 63:		Line 63:
	because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible ''state of memory'' it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.		because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible ''state of memory'' it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.

-	But there is another problem - [[Java]] is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the ''~~garbage collector~~ just kicks in'' and removes some "unneeded" objects from memory, which can change the behavior of the application - we used to have a never ending loop, which could be simulated only if two [[mozilla]] browsers and an evolution client was running as then the memory was small enough to invoke the ~~garbage collector~~. This kind of coverage is not measurable.	+	But there is another problem - [[Java]] is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the ''[[Garbage Collection]] just kicks in'' and removes some "unneeded" objects from memory, which can change the behavior of the application - we used to have a never ending loop, which could be simulated only if two [[mozilla]] browsers and an evolution client was running as then the memory was small enough to invoke the [[Garbage Collection]]. This kind of coverage is not measurable.

	That is why we suggest people to use code coverage tools as a way to sanity check that something is not really under tested. But it is necessary to remind ourselves that however high the coverage is, it does not prevent our application code fully from having bugs. So we, in order to help to fight the strange moves of an application [[amoeba]] shape, suggest to write a test when something gets broken - when there is a bug report, write a test to verify it and prevent regressions. That way the coverage is going to be focused on the code where it matters - the one that really was broken.		That is why we suggest people to use code coverage tools as a way to sanity check that something is not really under tested. But it is necessary to remind ourselves that however high the coverage is, it does not prevent our application code fully from having bugs. So we, in order to help to fight the strange moves of an application [[amoeba]] shape, suggest to write a test when something gets broken - when there is a bug report, write a test to verify it and prevent regressions. That way the coverage is going to be focused on the code where it matters - the one that really was broken.

JaroslavTulach: /* Systems are not Deterministic */

2018-10-20T05:12:50Z

Systems are not Deterministic

←Older revision		Revision as of 05:12, 20 October 2018
Line 22:		Line 22:
	==== Systems are not Deterministic ====		==== Systems are not Deterministic ====

-	Most of the critical systems are multi threaded. As such they are in a risk [[deadlock]]s or [[race condition]]s. Unless you test for that, you can't claim 100% coverage:	+	Most of the critical systems are multi threaded. As such they are in a risk of [[deadlock]]s or [[race condition]]s. Unless you test for that, you can't claim 100% coverage:
	* [[Java]] is not single threaded - [[GC]], finalizer, etc. run in parallel by default		* [[Java]] is not single threaded - [[GC]], finalizer, etc. run in parallel by default
	* [[JavaScript]] gets non-determinism via [[XMLHttpResource]], [[NodeJS]] is asynchronous (e.g. non-deterministic) by default		* [[JavaScript]] gets non-determinism via [[XMLHttpResource]], [[NodeJS]] is asynchronous (e.g. non-deterministic) by default

JaroslavTulach: /* All Function Arguments */

2018-10-20T05:12:04Z

All Function Arguments

←Older revision		Revision as of 05:12, 20 October 2018
Line 14:		Line 14:
	==== All Function Arguments ====		==== All Function Arguments ====

-	Nobody tests all possible arguments of a function. But unless ~~we do~~ it, we can't claim 100% coverage, because:	+	Nobody tests all possible arguments of a function. But unless it is done, we can't claim ''100% coverage'', because:
	* behavior of a function depends on all memory values it reads		* behavior of a function depends on all memory values it reads
	* all combination of read values should be covered		* all combination of read values should be covered

JaroslavTulach at 05:10, 20 October 2018

2018-10-20T05:10:57Z

←Older revision		Revision as of 05:10, 20 October 2018
Line 1:		Line 1:
-	"When there is enough [[wikipedia:Code_coverage]]?" was one of the questions asked by my 2005 post about [https://openide.netbeans.org/tutorial/test-patterns.html Test Patterns in Java]. Here is an extracted answer, prefaced by an executive summary.	+	"When there is enough [[wikipedia:Code_coverage\|code coverage]]?" was one of the questions asked by my 2005 post about [https://openide.netbeans.org/tutorial/test-patterns.html Test Patterns in Java]. Here is an extracted answer, prefaced by an executive summary.

	== There is no such thing as 100% [[CodeCoverage\|coverage]]! ==		== There is no such thing as 100% [[CodeCoverage\|coverage]]! ==

JaroslavTulach at 05:10, 20 October 2018

2018-10-20T05:10:40Z

←Older revision		Revision as of 05:10, 20 October 2018
Line 1:		Line 1:
-	When there is enough [[wikipedia:Code_coverage]]? ~~That is~~ one of the questions asked by my 2005 post about [https://openide.netbeans.org/tutorial/test-patterns.html Test Patterns in Java]. Here is an extracted answer.	+	"When there is enough [[wikipedia:Code_coverage]]?" was one of the questions asked by my 2005 post about [https://openide.netbeans.org/tutorial/test-patterns.html Test Patterns in Java]. Here is an extracted answer, prefaced by an executive summary.
		+
		+	== There is no such thing as 100% [[CodeCoverage\|coverage]]! ==
		+
		+	Whenever somebody starts arguing about [[CodeCoverage]] and especially measuring it, remember:
		+
		+	==== State of a program is wide ====
		+
		+	The state of a system/program is very wide. To claim one achieves 100% coverage, one needs to test it all. Hence repeat:
		+	* testing that each method is covered '''isn't enough'''
		+	* testing that each line is covered '''isn't enough'''
		+	* testing that each branch is visited '''isn't enough'''
		+
		+	==== All Function Arguments ====
		+
		+	Nobody tests all possible arguments of a function. But unless we do it, we can't claim 100% coverage, because:
		+	* behavior of a function depends on all memory values it reads
		+	* all combination of read values should be covered
		+	* repeating each test that deals with longs {{JDK\|java/lang\|Long}}.'''MAX_VALUE''' times would be pretty time consuming
		+	* nobody has time to test it all
		+
		+	==== Systems are not Deterministic ====
		+
		+	Most of the critical systems are multi threaded. As such they are in a risk [[deadlock]]s or [[race condition]]s. Unless you test for that, you can't claim 100% coverage:
		+	* [[Java]] is not single threaded - [[GC]], finalizer, etc. run in parallel by default
		+	* [[JavaScript]] gets non-determinism via [[XMLHttpResource]], [[NodeJS]] is asynchronous (e.g. non-deterministic) by default
		+	* simulating all possible race conditions is hard
		+
		+	=== Executive Summary ===
		+
		+	Please memorize and repeat all the time: ''there is no such thing as 100% coverage''!
		+

	== When there is enough tests? ==		== When there is enough tests? ==
Line 32:		Line 63:
	because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible ''state of memory'' it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.		because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible ''state of memory'' it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.

-	But there is another problem - [[Java]] is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the ''garbage collector just kicks in'' and removes some "unneeded" objects from memory, which can change the behavior of the application - we used to have a never ending loop, which could be simulated only if two [[mozilla]] browsers and an evolution client was running as then the memory was small enough to invoke the garbage collector. This kind of coverage is ~~unmeasurable~~.	+	But there is another problem - [[Java]] is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the ''garbage collector just kicks in'' and removes some "unneeded" objects from memory, which can change the behavior of the application - we used to have a never ending loop, which could be simulated only if two [[mozilla]] browsers and an evolution client was running as then the memory was small enough to invoke the garbage collector. This kind of coverage is not measurable.

	That is why we suggest people to use code coverage tools as a way to sanity check that something is not really under tested. But it is necessary to remind ourselves that however high the coverage is, it does not prevent our application code fully from having bugs. So we, in order to help to fight the strange moves of an application [[amoeba]] shape, suggest to write a test when something gets broken - when there is a bug report, write a test to verify it and prevent regressions. That way the coverage is going to be focused on the code where it matters - the one that really was broken.		That is why we suggest people to use code coverage tools as a way to sanity check that something is not really under tested. But it is necessary to remind ourselves that however high the coverage is, it does not prevent our application code fully from having bugs. So we, in order to help to fight the strange moves of an application [[amoeba]] shape, suggest to write a test when something gets broken - when there is a bug report, write a test to verify it and prevent regressions. That way the coverage is going to be focused on the code where it matters - the one that really was broken.

JaroslavTulach: /* When there is enough tests? */

2018-10-20T04:38:25Z

When there is enough tests?

←Older revision		Revision as of 04:38, 20 October 2018
Line 5:		Line 5:
	While writing tests, people can ask: how many of them should be written? The simple answer is to write tests while they are useful. The more precise, more complex and less clear answer is going to be covered in this chapter.		While writing tests, people can ask: how many of them should be written? The simple answer is to write tests while they are useful. The more precise, more complex and less clear answer is going to be covered in this chapter.

-	There are various tools out there that help to measure test coverage. [[NetBeans]] project selected ''emma'' for measuring the coverage of our application code by our tests. When invoked ~~(for example from the popup menu of any project from NetBeans.org)~~ it instruments the application code and invokes automated tests on it. While running, it collects information about all called methods, visited classes and lines and then it shows a summary in a web browser.	+	There are various tools out there that help to measure test coverage. [[NetBeans]] project selected ''emma'' for measuring the coverage of our application code by our tests. When invoked it instruments the application code and invokes automated tests on it. While running, it collects information about all called methods, visited classes and lines and then it shows a summary in a web browser.

	Counting coverage by visited methods is very rough criteria, but it can be surprisingly hard to get close to 100%. But even if you succeed, there is no guarantee that the resulting application code works correctly. Every methods has a few input parameters, and knowing that it succeeded once with one selection of them, does not say anything about the other cases.		Counting coverage by visited methods is very rough criteria, but it can be surprisingly hard to get close to 100%. But even if you succeed, there is no guarantee that the resulting application code works correctly. Every methods has a few input parameters, and knowing that it succeeded once with one selection of them, does not say anything about the other cases.

JaroslavTulach at 13:48, 4 April 2018

2018-04-04T13:48:25Z

←Older revision		Revision as of 13:48, 4 April 2018
Line 1:		Line 1:
-	When there is enough [[wikipedia:Code_coverage]]? That is ~~a question~~ asked by my 2005 post about [https://openide.netbeans.org/tutorial/test-patterns.html Test Patterns in Java].	+	When there is enough [[wikipedia:Code_coverage]]? That is one of the questions asked by my 2005 post about [https://openide.netbeans.org/tutorial/test-patterns.html Test Patterns in Java]. Here is an extracted answer.

	== When there is enough tests? ==		== When there is enough tests? ==

JaroslavTulach: /* When there is enough tests? */

2018-04-03T11:55:47Z

When there is enough tests?

←Older revision		Revision as of 11:55, 3 April 2018
Line 30:		Line 30:
	percentage(5);		percentage(5);
	</source>		</source>
-	because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible state of memory it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.	+	because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible ''state of memory'' it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.

-	But there is another problem - Java is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the garbage collector just kicks in and removes some "unneeded" objects from memory, which can change the ~~behaviour~~ of the application - we used to have a never ending loop, which could be simulated only if two mozilla browsers and an evolution client was running as then the memory was small enough to invoke the garbage collector. This kind of coverage is unmeasurable.	+	But there is another problem - [[Java]] is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the ''garbage collector just kicks in'' and removes some "unneeded" objects from memory, which can change the behavior of the application - we used to have a never ending loop, which could be simulated only if two [[mozilla]] browsers and an evolution client was running as then the memory was small enough to invoke the garbage collector. This kind of coverage is unmeasurable.

	That is why we suggest people to use code coverage tools as a way to sanity check that something is not really under tested. But it is necessary to remind ourselves that however high the coverage is, it does not prevent our application code fully from having bugs. So we, in order to help to fight the strange moves of an application [[amoeba]] shape, suggest to write a test when something gets broken - when there is a bug report, write a test to verify it and prevent regressions. That way the coverage is going to be focused on the code where it matters - the one that really was broken.		That is why we suggest people to use code coverage tools as a way to sanity check that something is not really under tested. But it is necessary to remind ourselves that however high the coverage is, it does not prevent our application code fully from having bugs. So we, in order to help to fight the strange moves of an application [[amoeba]] shape, suggest to write a test when something gets broken - when there is a bug report, write a test to verify it and prevent regressions. That way the coverage is going to be focused on the code where it matters - the one that really was broken.

JaroslavTulach: /* When there is enough tests? */

2018-04-03T11:53:27Z

When there is enough tests?

←Older revision		Revision as of 11:53, 3 April 2018
Line 9:		Line 9:
	Counting coverage by visited methods is very rough criteria, but it can be surprisingly hard to get close to 100%. But even if you succeed, there is no guarantee that the resulting application code works correctly. Every methods has a few input parameters, and knowing that it succeeded once with one selection of them, does not say anything about the other cases.		Counting coverage by visited methods is very rough criteria, but it can be surprisingly hard to get close to 100%. But even if you succeed, there is no guarantee that the resulting application code works correctly. Every methods has a few input parameters, and knowing that it succeeded once with one selection of them, does not say anything about the other cases.

-	Much better is to count the coverage by branches or lines. When there is a if (...) { x(); } else { y(); } statement in code of your method, you want to be sure that both methods, x and y will be called. The emma tool supports this and by helping us to be sure that every line is visited, it gives us confidence that our application code does not contain useless lines.	+	Much better is to count the coverage by branches or lines. When there is a
		+	<source lang="java">
		+	if (...) { x(); } else { y(); }
		+	</source>
		+	statement in code of your method, you want to be sure that both methods, x and y will be called. The emma tool supports this and by helping us to be sure that every line is visited, it gives us confidence that our application code does not contain useless lines.

	Still, the fact that a line is visited once, does not mean that our application code is not buggy.		Still, the fact that a line is visited once, does not mean that our application code is not buggy.
Line 21:		Line 25:
	}		}
	</source>		</source>
-	It is good if both methods get executed, and fine if we test them with various parameters - still we can get an error if we call add (-10); percentage(5), because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible state of memory it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.	+	It is good if both methods get executed, and fine if we test them with various parameters - still we can get an error if we call
		+	<source lang="java">
		+	add (-10);
		+	percentage(5);
		+	</source>
		+	because the sum will be zero and division by zero is forbidden. To be sure that our application is not vulnerable to problems like this, we would have to test each method in each possible state of memory it depends on (e.g. each value of sum variable) and that would give us the ultimate proof that our application code works correctly in a single threaded environment.

	But there is another problem - Java is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the garbage collector just kicks in and removes some "unneeded" objects from memory, which can change the behaviour of the application - we used to have a never ending loop, which could be simulated only if two mozilla browsers and an evolution client was running as then the memory was small enough to invoke the garbage collector. This kind of coverage is unmeasurable.		But there is another problem - Java is not single threaded. A lot of applications start new threads by themselves, and even if they do not, there is the AWT event dispatch thread, the finalizer thread, etc. So one has to count on some amount of non-determinism. Sometimes the garbage collector just kicks in and removes some "unneeded" objects from memory, which can change the behaviour of the application - we used to have a never ending loop, which could be simulated only if two mozilla browsers and an evolution client was running as then the memory was small enough to invoke the garbage collector. This kind of coverage is unmeasurable.

JaroslavTulach: /* When there is enough tests? */

2018-04-03T11:51:26Z

When there is enough tests?

←Older revision		Revision as of 11:51, 3 April 2018
Line 5:		Line 5:
	While writing tests, people can ask: how many of them should be written? The simple answer is to write tests while they are useful. The more precise, more complex and less clear answer is going to be covered in this chapter.		While writing tests, people can ask: how many of them should be written? The simple answer is to write tests while they are useful. The more precise, more complex and less clear answer is going to be covered in this chapter.

-	There are various tools out there that help to measure test coverage. ~~We have~~ selected emma for measuring the coverage of our application code by our tests. When invoked (for example from the popup menu of any project from NetBeans.org) it instruments the application code and invokes automated tests on it. While running, it collects information about all called methods, visited classes and lines and then it shows a summary in a web browser.	+	There are various tools out there that help to measure test coverage. [[NetBeans]] project selected ''emma'' for measuring the coverage of our application code by our tests. When invoked (for example from the popup menu of any project from NetBeans.org) it instruments the application code and invokes automated tests on it. While running, it collects information about all called methods, visited classes and lines and then it shows a summary in a web browser.

	Counting coverage by visited methods is very rough criteria, but it can be surprisingly hard to get close to 100%. But even if you succeed, there is no guarantee that the resulting application code works correctly. Every methods has a few input parameters, and knowing that it succeeded once with one selection of them, does not say anything about the other cases.		Counting coverage by visited methods is very rough criteria, but it can be surprisingly hard to get close to 100%. But even if you succeed, there is no guarantee that the resulting application code works correctly. Every methods has a few input parameters, and knowing that it succeeded once with one selection of them, does not say anything about the other cases.