SignatureTest

From APIDesign

Copy of netbeans:SignatureTest created for archiving purposes.

Signature tests verify that public and protected members of non-private classes in each module do not change or remain compatible with previous version. These tests are part of the build or the continuous build process. The testing infrastructure is currently based on APITest, an extension and temporary fork of sigtest tool which is used as part of the JCP test compatibility kits to verify that the API exposed by a JSR implementation does not diverge from the reference implementation.

Use from Maven

See SigTest for the details.

Consistency Check

Turned on Nov 7, 2012:

For a while the VersioningPolicy prescribes how to properly version modules and their API changes. However the policy has for a long time only been applied to modules with stable API. From time to time this leads to no such method or class errors like Template:Iz which need to be evaluated by platform/module system component owner.

Moreover the introduction of incremental binary build AutoUpdateTask requires more proper attitude to versioning. Only then one can be sure that incremental binary build really links properly.

For the above reasons ita consistency check that verifies minimal versioning consistency has been introduced. The HgParallelProjectIntegration runs:

Check on all modules exposing non-private packages
Compares the spec version.

If the new spec is the same as previous, the check fails if there is some API change (method or class)
If the new spec is higher, the check will pass regardless of any API changes.

The check is integral part of push-* jobs and if it detects a failure, the push job fails. To recover you are supposed to increment spec version of your module whenever you change its API. Increment spec versions of modules that depend on your API change as well (not enforced right now). Run following command to simulate the check locally:

# first of all make sure your system is built OK
# Then either:
# generate appropriate JUnit testing report for all modules:
$ ant check-sigtests-version 
#
# or run the 'version' check for a single module
$ ant -f your.module/build.xml check-sigtest -Dsigtest.check.type=versioncheck

Continuous Check for Changes

The continuous build process runs a sigtest build that compares APIs in a new build with APIs of the one of the previous build. Whenever there is a change (either compatible or incompatible) a mail is sent:

if the module has public packages - to the api-changes@netbeans.org mailing list
if the module has friends in other clusters - to the api-changes@netbeans.org mailing list
if the module has friends only in the same cluster - to appropriate cvs@<module>.netbeans.org mailing list
if the module has neither friends, neither public packages - it is skipped from the check

Appropriate owners of each module and tech leads of each area are expected to review the change and prevent or revert unintentional changes.

The signature golden files are generated by

$ ant -f nbbuild/build.xml gen-sigtests

and compared by

$ ant -f nbbuild/build.xml gen-sigtests

A module can configure sigtest behavior by modifying its project.properties:

# redefine the email where reports are sent
sigtest.mail=your.name@your.org
# don't fail if the generation of the report fails
sigtest.gen.fail.on.error=false

At the time of release

When doing a release it is important to record the actual signatures of stable modules (and to some extend also under development ones), so we know what to keep [[apidesign::BackwardCompatibility|backward compatibility]] with. To generate these signatures we used to do following, but since Apache NetBeans is in Git, it is better to follow the PR-1064 steps. Anyway here is the old recipe:

$ hg update -C releaseXY_fcs
$ ant clean
$ ant build
$ ant gen-sigtests-release
$ hg st|cut -c 3-| xargs zip /tmp/sig.zip

The last target will modify a lot of module/nbproject/*.sig files and possibly generate some new. These files shall be integrated into trunk build. For example by:

$ cd <team-repository>
$ hg update -c default
$ unzip /tmp/sig.zip
$ hg add .
$ hg ci -m "Signatures as of release X.Y"
$ hg push

Check for Incompatible Changes Since Previous Version

There is another check for API signature compliance: A check that the APIs are compatible with a previous released version. This check is available to every module, which can make it part of its build by generating its API snapshot by:

ant netbeans gen-sigtest

This creates a file nbproject/org-netbeans-module-name.sig which can be checked into the version control system as a reference point. This check is part of continuous build as well: every module with its golden file in nbproject/org-netbeans-module-name.sig is checked for backward compatibility and failures are included in continuous build test results. To run the same test for one module on local computer just use:

$ ant netbeans check-sigtest

in your module directory and API of your module will be verified. Of course, only if the signature file exists.

Who needs the Golden Signature File

Not every module with an API needs this golden file; currently only those modules that produced a stable API in release 6.1 have it. The plan is to update these golden files with a new version as soon as it is released, so future checks are done against latest released version.

If an incompatible change is made (discouraged and subject to API review process) the golden file needs to be edited by hand to remove the element that is no longer present in the API.

Checking for all changes from previous release

In case you have the sig file in nbproject directory generated at the time of previour release (all public APIs have it), you can see all changes since that time by:

$ ant -f yourmodule/build.xml check-sigtest -Dsigtest.check.type=strictcheck

apitest tool

This check for incompatibilities is done by NetBeans' own version of an OpenJDK tools. This adds a standard Ant task to check for binary backward compatibility, mutual signature compatibility. The binary compatibility check ignores generic types. Its sources are distributed together with its Mercurial repository and are available under GPL version 2. Builds are available from our hudson builder.