New page: Copy of netbeans:SignatureTest created for archiving purposes. __NOTOC__ Signature tests verify that public and protected members of non-private classes in each module do not change o...
New page
Copy of [[netbeans:SignatureTest]] created for archiving purposes.
__NOTOC__
Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the [http://deadlock.netbeans.org/hudson/ continuous build process]. The testing infrastructure is currently based on [[APITest | APITest]], an extension and temporary fork of [http://sigtest.java.net/ sigtest tool] which is used as part of the [http://www.jcp.org/ JCP] test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.
== Use from Maven ==
See [[SigTest]] for the details.
== Consistency Check ==
'''Turned on Nov 7, 2012''':
For a while the [[VersioningPolicy]] prescribes how to properly version modules and their [[API]] changes. However the policy has for a long time only been applied to modules with stable [[API]]. From time to time this leads to ''no such method or class'' errors like {{iz|186233}} which need to be evaluated by ''platform/module system'' component owner.
Moreover the introduction of incremental binary build [[AutoUpdateTask]] requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.
For the above reasons ita consistency check that verifies minimal versioning consistency has been introduced. The [[HgParallelProjectIntegration]] runs:
* Check on all modules exposing non-private packages
* Compares the spec version.
:* If the new spec is the same as previous, the check fails if there is some [[API]] change (method or class)
:* If the new spec is higher, the check will pass regardless of any [[API]] changes.
The check is integral part of push-* jobs and if it detects a failure, the push job fails. '''To recover''' you are supposed to increment spec version of your module whenever you change its API. Increment spec versions of modules that depend on your [[API]] change as well (not enforced right now). Run following command to simulate the check locally:
<source lang="bash">
# first of all make sure your system is built OK
# Then either:
# generate appropriate JUnit testing report for all modules:
$ ant check-sigtests-version
#
# or run the 'version' check for a single module
$ ant -f your.module/build.xml check-sigtest -Dsigtest.check.type=versioncheck
</source>
<!--
In case there is a strong need to disable this kind of version checking for an individual module (provided that the owners know how to version properly), one can add following into own ''nbproject/project.properties'' file: ''sigtest.check.type=check''. This is usefull mostly for modules with split packages.
-->
==Continuous Check for Changes==
The [http://deadlock.netbeans.org/hudson/ continuous build process] runs a sigtest build that compares APIs in a new build with APIs of the one of the previous build. Whenever there is a change (either compatible or incompatible) a mail is sent:
* if the module has public packages - to the [mailto:api-changes@netbeans.org api-changes@netbeans.org] mailing list
* if the module has friends in other clusters - to the [mailto:api-changes@netbeans.org api-changes@netbeans.org] mailing list
* if the module has friends only in the same cluster - to appropriate '''cvs@<module>.netbeans.org''' mailing list
* if the module has neither friends, neither public packages - it is skipped from the check
Appropriate owners of each module and tech leads of each area are expected to review the change and prevent or revert unintentional changes.
The [http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/sigtest/snapshot/ signature golden files] are generated by
<source lang="bash">$ ant -f nbbuild/build.xml gen-sigtests</source>
and compared by
<source lang="bash">$ ant -f nbbuild/build.xml gen-sigtests</source>
A module can configure sigtest behavior by modifying its '''project.properties''':
<source lang="bash">
# redefine the email where reports are sent
sigtest.mail=your.name@your.org
# don't fail if the generation of the report fails
sigtest.gen.fail.on.error=false
</source>
== At the time of release ==
When doing a release it is important to record the actual signatures of ''stable'' modules (and to some extend also ''under development'' ones), so we know what to keep [[apidesign::BackwardCompatibility|backward compatibility]] with. To generate these signatures we used to do following, but since [https://github.com/apache/incubator-netbeans/pull/1064 Apache NetBeans is in Git], it is better to follow the [https://github.com/apache/incubator-netbeans/pull/1064 PR-1064 steps]. Anyway here is the old recipe:
<source lang="bash">
$ hg update -C releaseXY_fcs
$ ant clean
$ ant build
$ ant gen-sigtests-release
$ hg st|cut -c 3-| xargs zip /tmp/sig.zip
</source>
The last target will modify a lot of module/nbproject/*.sig files and possibly generate some new. These files shall be integrated into trunk build. For example by:
<source lang="bash">
$ cd <team-repository>
$ hg update -c default
$ unzip /tmp/sig.zip
$ hg add .
$ hg ci -m "Signatures as of release X.Y"
$ hg push
</source>
==Check for Incompatible Changes Since Previous Version==
There is another check for API signature compliance: A check that the APIs are compatible with a previous released version. This check is available to every module, which can make it part of its build by generating its API snapshot by:
<source lang="bash">
ant netbeans gen-sigtest
</source>
This creates a file nbproject/org-netbeans-module-name.sig which can be checked into the version control system as a reference point. This check is part of continuous build as well: every module with its golden file in nbproject/org-netbeans-module-name.sig is checked for backward compatibility and failures are included in continuous build [http://deadlock.netbeans.org/hudson/job/trunk/lastBuild/testReport/ test results]. To run the same test for one module on local computer just use:
<source lang="bash">
$ ant netbeans check-sigtest
</source>
in your module directory and API of your module will be verified. Of course, only if the signature file exists.
===Who needs the Golden Signature File===
Not every module with an API needs this golden file; currently only those modules that produced a stable API in release 6.1 have it. The plan is to update these golden files with a new version as soon as it is released, so future checks are done against latest released version.
If an incompatible change is made (discouraged and subject to [http://openide.netbeans.org/tutorial/reviews API review process]) the golden file needs to be edited by hand to remove the element that is no longer present in the API.
== Checking for all changes from previous release ==
In case you have the sig file in nbproject directory generated at the time of previour release (all public APIs have it), you can see all changes since that time by:
<source lang="bash">
$ ant -f yourmodule/build.xml check-sigtest -Dsigtest.check.type=strictcheck
</source>
===apitest tool===
This check for incompatibilities is done by [[APITest| NetBeans' own version]] of an OpenJDK tools. This adds a standard Ant task to check for binary backward compatibility, mutual signature compatibility. The binary compatibility check ignores generic types. Its [http://hg.netbeans.org/apitest/ sources] are distributed together with its Mercurial repository and are available under GPL version 2. Builds are available from our [http://deadlock.netbeans.org/hudson/job/apitest/ hudson builder].
===Other NetBeans Tests===
* [[FindBugsTest | FindBugsTest]]
__NOTOC__
Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the [http://deadlock.netbeans.org/hudson/ continuous build process]. The testing infrastructure is currently based on [[APITest | APITest]], an extension and temporary fork of [http://sigtest.java.net/ sigtest tool] which is used as part of the [http://www.jcp.org/ JCP] test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.
== Use from Maven ==
See [[SigTest]] for the details.
== Consistency Check ==
'''Turned on Nov 7, 2012''':
For a while the [[VersioningPolicy]] prescribes how to properly version modules and their [[API]] changes. However the policy has for a long time only been applied to modules with stable [[API]]. From time to time this leads to ''no such method or class'' errors like {{iz|186233}} which need to be evaluated by ''platform/module system'' component owner.
Moreover the introduction of incremental binary build [[AutoUpdateTask]] requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.
For the above reasons ita consistency check that verifies minimal versioning consistency has been introduced. The [[HgParallelProjectIntegration]] runs:
* Check on all modules exposing non-private packages
* Compares the spec version.
:* If the new spec is the same as previous, the check fails if there is some [[API]] change (method or class)
:* If the new spec is higher, the check will pass regardless of any [[API]] changes.
The check is integral part of push-* jobs and if it detects a failure, the push job fails. '''To recover''' you are supposed to increment spec version of your module whenever you change its API. Increment spec versions of modules that depend on your [[API]] change as well (not enforced right now). Run following command to simulate the check locally:
<source lang="bash">
# first of all make sure your system is built OK
# Then either:
# generate appropriate JUnit testing report for all modules:
$ ant check-sigtests-version
#
# or run the 'version' check for a single module
$ ant -f your.module/build.xml check-sigtest -Dsigtest.check.type=versioncheck
</source>
<!--
In case there is a strong need to disable this kind of version checking for an individual module (provided that the owners know how to version properly), one can add following into own ''nbproject/project.properties'' file: ''sigtest.check.type=check''. This is usefull mostly for modules with split packages.
-->
==Continuous Check for Changes==
The [http://deadlock.netbeans.org/hudson/ continuous build process] runs a sigtest build that compares APIs in a new build with APIs of the one of the previous build. Whenever there is a change (either compatible or incompatible) a mail is sent:
* if the module has public packages - to the [mailto:api-changes@netbeans.org api-changes@netbeans.org] mailing list
* if the module has friends in other clusters - to the [mailto:api-changes@netbeans.org api-changes@netbeans.org] mailing list
* if the module has friends only in the same cluster - to appropriate '''cvs@<module>.netbeans.org''' mailing list
* if the module has neither friends, neither public packages - it is skipped from the check
Appropriate owners of each module and tech leads of each area are expected to review the change and prevent or revert unintentional changes.
The [http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/build/sigtest/snapshot/ signature golden files] are generated by
<source lang="bash">$ ant -f nbbuild/build.xml gen-sigtests</source>
and compared by
<source lang="bash">$ ant -f nbbuild/build.xml gen-sigtests</source>
A module can configure sigtest behavior by modifying its '''project.properties''':
<source lang="bash">
# redefine the email where reports are sent
sigtest.mail=your.name@your.org
# don't fail if the generation of the report fails
sigtest.gen.fail.on.error=false
</source>
== At the time of release ==
When doing a release it is important to record the actual signatures of ''stable'' modules (and to some extend also ''under development'' ones), so we know what to keep [[apidesign::BackwardCompatibility|backward compatibility]] with. To generate these signatures we used to do following, but since [https://github.com/apache/incubator-netbeans/pull/1064 Apache NetBeans is in Git], it is better to follow the [https://github.com/apache/incubator-netbeans/pull/1064 PR-1064 steps]. Anyway here is the old recipe:
<source lang="bash">
$ hg update -C releaseXY_fcs
$ ant clean
$ ant build
$ ant gen-sigtests-release
$ hg st|cut -c 3-| xargs zip /tmp/sig.zip
</source>
The last target will modify a lot of module/nbproject/*.sig files and possibly generate some new. These files shall be integrated into trunk build. For example by:
<source lang="bash">
$ cd <team-repository>
$ hg update -c default
$ unzip /tmp/sig.zip
$ hg add .
$ hg ci -m "Signatures as of release X.Y"
$ hg push
</source>
==Check for Incompatible Changes Since Previous Version==
There is another check for API signature compliance: A check that the APIs are compatible with a previous released version. This check is available to every module, which can make it part of its build by generating its API snapshot by:
<source lang="bash">
ant netbeans gen-sigtest
</source>
This creates a file nbproject/org-netbeans-module-name.sig which can be checked into the version control system as a reference point. This check is part of continuous build as well: every module with its golden file in nbproject/org-netbeans-module-name.sig is checked for backward compatibility and failures are included in continuous build [http://deadlock.netbeans.org/hudson/job/trunk/lastBuild/testReport/ test results]. To run the same test for one module on local computer just use:
<source lang="bash">
$ ant netbeans check-sigtest
</source>
in your module directory and API of your module will be verified. Of course, only if the signature file exists.
===Who needs the Golden Signature File===
Not every module with an API needs this golden file; currently only those modules that produced a stable API in release 6.1 have it. The plan is to update these golden files with a new version as soon as it is released, so future checks are done against latest released version.
If an incompatible change is made (discouraged and subject to [http://openide.netbeans.org/tutorial/reviews API review process]) the golden file needs to be edited by hand to remove the element that is no longer present in the API.
== Checking for all changes from previous release ==
In case you have the sig file in nbproject directory generated at the time of previour release (all public APIs have it), you can see all changes since that time by:
<source lang="bash">
$ ant -f yourmodule/build.xml check-sigtest -Dsigtest.check.type=strictcheck
</source>
===apitest tool===
This check for incompatibilities is done by [[APITest| NetBeans' own version]] of an OpenJDK tools. This adds a standard Ant task to check for binary backward compatibility, mutual signature compatibility. The binary compatibility check ignores generic types. Its [http://hg.netbeans.org/apitest/ sources] are distributed together with its Mercurial repository and are available under GPL version 2. Builds are available from our [http://deadlock.netbeans.org/hudson/job/apitest/ hudson builder].
===Other NetBeans Tests===
* [[FindBugsTest | FindBugsTest]]