TruffleSigtest
From APIDesign
(→Setting the Snapshots Up) |
(→Notification of Daily Changes) |
||
(15 intermediate revisions not shown.) | |||
Line 3: | Line 3: | ||
==== Identify Your API ==== | ==== Identify Your API ==== | ||
- | [[ | + | Not all [[Java]] packages in your codebase are an [[API]]! Make sure you identify what is and what isn't part of an [[API]]. This is useful for packaging (for example to include correct [[OSGi]] metadata in manifests of your [[JAR]] files), generating [[Javadoc]] (see the [http://lafo.ssw.uni-linz.ac.at/javadoc/truffle/latest/ Truffle documentation]) and also for [[SignatureTests|signature testing]] - you don't want to be notified about every change in implementation packages, you care only about the [[API]] parts. |
+ | |||
+ | Each project uses its own way to identify the [[API]] packages. The [[Truffle]] project decided to use '''package-info.java''' - if there is a '''package-info.java''' file in a directory, its (public) classes are part of the [[API]], if not, they are an implementation detail. | ||
==== The Right Tool ==== | ==== The Right Tool ==== | ||
Line 15: | Line 17: | ||
==== Integrate into Build Process ==== | ==== Integrate into Build Process ==== | ||
- | [[SignatureTests|Signature testing]] needs to be integrated into build process of your project - in case there is a violation with respect to [[BackwardCompatibility]], the build should fail. As (almost) every big project | + | [[SignatureTests|Signature testing]] needs to be integrated into build process of your project - in case there is a violation with respect to [[BackwardCompatibility]], the build should fail. As (almost) every big project invents its own build harness, the ways to integrate differ. For example, [[NetBeans]] is using [[Ant]] based harness and thus one deals with the signature testing tool via [[Ant]] as described at [[netbeans:SignatureTest]] page. |
The [[Truffle]] project (as well as [[Graal]]) is using [[Python]] based harness. As such [[I]] had to write a bit of [[Python]] code to integrate the [[netbeans:APITest|APITest]] into it (see [https://bitbucket.org/allr/mx/src/e7fa425baca686c418ffa3cb521fa3f853a994a6/mx_sigtest.py here]). You basically need two operations: | The [[Truffle]] project (as well as [[Graal]]) is using [[Python]] based harness. As such [[I]] had to write a bit of [[Python]] code to integrate the [[netbeans:APITest|APITest]] into it (see [https://bitbucket.org/allr/mx/src/e7fa425baca686c418ffa3cb521fa3f853a994a6/mx_sigtest.py here]). You basically need two operations: | ||
# generate - which will take a snapshot of your [[API]] at a certain point of time (usually at a release time) | # generate - which will take a snapshot of your [[API]] at a certain point of time (usually at a release time) | ||
# check - which will compare the current state of your [[API]] with the existing snapshot | # check - which will compare the current state of your [[API]] with the existing snapshot | ||
- | + | To invoke these operations in [[Truffle]] repository use: | |
<source lang="bash"> | <source lang="bash"> | ||
$ mx sigtest --generate | $ mx sigtest --generate | ||
Line 39: | Line 41: | ||
[[I]]'ve integrated the "--check binary" into so called ''gate check'', so if there is an incompatible change, the build fails - see the [http://lafo.ssw.uni-linz.ac.at/hg/truffle/diff/b07854a19ab4/mx.truffle/mx_truffle.py changeset]. | [[I]]'ve integrated the "--check binary" into so called ''gate check'', so if there is an incompatible change, the build fails - see the [http://lafo.ssw.uni-linz.ac.at/hg/truffle/diff/b07854a19ab4/mx.truffle/mx_truffle.py changeset]. | ||
- | |||
==== Setting the Snapshots Up ==== | ==== Setting the Snapshots Up ==== | ||
Line 47: | Line 48: | ||
<source lang="bash"> | <source lang="bash"> | ||
$ hg up -C truffle-0.9 | $ hg up -C truffle-0.9 | ||
+ | $ mx build | ||
$ mx sigtest --generate | $ mx sigtest --generate | ||
+ | $ hg up default | ||
$ hg add . | $ hg add . | ||
$ hg ci -m "Adding API snapshots as for version truffle-0.9 and enabling their check in the gate" | $ hg ci -m "Adding API snapshots as for version truffle-0.9 and enabling their check in the gate" | ||
Line 53: | Line 56: | ||
Here is how such snapshot files look like: [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/b07854a19ab4 changeset]. Of course, it turned out that not everything is perfect (how it could be! we weren't running the tests and it is so hard to reason about [[BackwardCompatibility]] without testing): | Here is how such snapshot files look like: [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/b07854a19ab4 changeset]. Of course, it turned out that not everything is perfect (how it could be! we weren't running the tests and it is so hard to reason about [[BackwardCompatibility]] without testing): | ||
- | * Removing API elements that were deprecated at the time of truffle-0.9 release from the list of required API elements: [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/2ce4c49bc131 2ce4c49bc131] - [[Truffle]] project is still in its early phases, so according to our [http://lafo.ssw.uni-linz.ac.at/javadoc/truffle/latest/overview-summary.html compatibility policy] | + | * Removing API elements that were deprecated at the time of truffle-0.9 release from the list of required API elements: [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/2ce4c49bc131 2ce4c49bc131] - [[Truffle]] project is still in its early phases, so according to our [http://lafo.ssw.uni-linz.ac.at/javadoc/truffle/latest/overview-summary.html compatibility policy] removing deprecated elements is OK. Once we reach release 1.0, we won't do things like this. |
* The check discovered a removed constructor [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/5033b980cc68 5033b980cc68] which was clearly a bug | * The check discovered a removed constructor [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/5033b980cc68 5033b980cc68] which was clearly a bug | ||
* Adjusting to binary incompatible change of (on)returnExceptional parameters [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/7273b139fff2 7273b139fff2] - this was an intentional and agreed on change (in spite of being against the [http://lafo.ssw.uni-linz.ac.at/javadoc/truffle/latest/overview-summary.html compatibility policy]) | * Adjusting to binary incompatible change of (on)returnExceptional parameters [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/7273b139fff2 7273b139fff2] - this was an intentional and agreed on change (in spite of being against the [http://lafo.ssw.uni-linz.ac.at/javadoc/truffle/latest/overview-summary.html compatibility policy]) | ||
* And at last, there was an [[API]] [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/23d2b5513c83 design problem] - a return type of an [[API]] visible method was not public - e.g. visible | * And at last, there was an [[API]] [http://lafo.ssw.uni-linz.ac.at/hg/truffle/rev/23d2b5513c83 design problem] - a return type of an [[API]] visible method was not public - e.g. visible | ||
- | In all of these cases [[I]] manually updated the signature files to match the expected state. Then the compatibility check finally succeeded. | + | In all of these cases [[I]] manually updated the signature files to match the expected state. Then the compatibility check finally succeeded. Since today it will be way easier to keep [[BackwardCompatibility]] in our [[Truffle]] project! |
+ | |||
+ | ==== Notification of Daily Changes ==== | ||
+ | |||
+ | In addition to catching [[BackwardCompatible|incompatible]] changes, it is useful to track [[BackwardCompatible|compatible ones]] as well. | ||
+ | |||
+ | Some of the [[API]] changes happen by an accident and when they aren't noticed in time, they may become part of a release. But then they have to stay (as they are like [[star]]s) regardless how bad they are. Nobody wants to maintain erroneous code or [[API]], so it is better to catch such changes as soon as possible and revert them before they propagate to a release. | ||
+ | |||
+ | Having a [[Hudson]] job to watch daily compatible changes and notify us about them is the best way to prevent accidental [[API]] changes. The job shall have two parts: generate new snapshot of today's [[API]]s: | ||
+ | |||
+ | <source lang="bash"> | ||
+ | $ mx sigtest --generate | ||
+ | $ zip snapshots.zip truffle/*/*sigtest | ||
+ | </source> | ||
+ | |||
+ | and make them persisted artifacts of the job. The other part is to take these artifacts from the last successful build and compare them against current state: | ||
+ | |||
+ | <source lang="bash"> | ||
+ | $ wget $HUDSON_URL/lastSuccessfulBuild/snaphots.zip | ||
+ | $ unzip snapshots.zip | ||
+ | $ mx sigtest --check all | ||
+ | </source> | ||
+ | |||
+ | then it is just a matter of sending the output of the last command to an observer mailing list, so humans can decide whether the change is intended or not. | ||
+ | |||
+ | [[I]] believe that with infrastructure like this the [[Truffle]] project will be able to stick to its [http://lafo.ssw.uni-linz.ac.at/javadoc/truffle/latest/overview-summary.html promise of compatibility] and better preserve investments done by [[language]] writers porting their languages to the [[RubySpeed|fastest (J)VM on the Planet]]! |
Current revision
API signature tests are part of Truffle project workflow since today! Let me describe what I have done, as similar changes are likely needed in any Java project that wants to use SignatureTests to keep backward compatibility of its APIs.
Contents |
Identify Your API
Not all Java packages in your codebase are an API! Make sure you identify what is and what isn't part of an API. This is useful for packaging (for example to include correct OSGi metadata in manifests of your JAR files), generating Javadoc (see the Truffle documentation) and also for signature testing - you don't want to be notified about every change in implementation packages, you care only about the API parts.
Each project uses its own way to identify the API packages. The Truffle project decided to use package-info.java - if there is a package-info.java file in a directory, its (public) classes are part of the API, if not, they are an implementation detail.
The Right Tool
First and foremost one needs to select a signature testing tool. As my background is related to NetBeans, my choice was almost guaranteed. I decided to use NetBeans APITest.
Slight problem was that the tool hasn't been updated to JDK8 language constructs (e.g. default and static methods in interfaces). However, as NetBeans need to use the tool as well, I got a great help from Tomáš Zezula. He is currently working on switching the NetBeans codebase to JDK8 and he updated the APITest to deal properly with JDK8 features (see here and here). It is always good to build on shoulders of giants - e.g. use a tool that a mature project like NetBeans maintains.
Download the APITest from here.
Integrate into Build Process
Signature testing needs to be integrated into build process of your project - in case there is a violation with respect to BackwardCompatibility, the build should fail. As (almost) every big project invents its own build harness, the ways to integrate differ. For example, NetBeans is using Ant based harness and thus one deals with the signature testing tool via Ant as described at netbeans:SignatureTest page.
The Truffle project (as well as Graal) is using Python based harness. As such I had to write a bit of Python code to integrate the APITest into it (see here). You basically need two operations:
- generate - which will take a snapshot of your API at a certain point of time (usually at a release time)
- check - which will compare the current state of your API with the existing snapshot
To invoke these operations in Truffle repository use:
$ mx sigtest --generate # generates following files truffle/com.oracle.truffle.api.dsl/snapshot.sigtest truffle/com.oracle.truffle.api.interop.java/snapshot.sigtest truffle/com.oracle.truffle.api.interop/snapshot.sigtest truffle/com.oracle.truffle.api.object/snapshot.sigtest truffle/com.oracle.truffle.api/snapshot.sigtest truffle/com.oracle.truffle.api.vm/snapshot.sigtest # check whether there are any incompatible changes $ mx sigtest --check binary # check if there are any API changes $ mx sigtest --check all
I've integrated the "--check binary" into so called gate check, so if there is an incompatible change, the build fails - see the changeset.
Setting the Snapshots Up
BackwardCompatibility is important between releases, so once you do a release (whatever that means), generate a snapshot of your API state. The most recent (e.g. JavaOne 2015) release of Truffle is 0.9, so I just did:
$ hg up -C truffle-0.9 $ mx build $ mx sigtest --generate $ hg up default $ hg add . $ hg ci -m "Adding API snapshots as for version truffle-0.9 and enabling their check in the gate"
Here is how such snapshot files look like: changeset. Of course, it turned out that not everything is perfect (how it could be! we weren't running the tests and it is so hard to reason about BackwardCompatibility without testing):
- Removing API elements that were deprecated at the time of truffle-0.9 release from the list of required API elements: 2ce4c49bc131 - Truffle project is still in its early phases, so according to our compatibility policy removing deprecated elements is OK. Once we reach release 1.0, we won't do things like this.
- The check discovered a removed constructor 5033b980cc68 which was clearly a bug
- Adjusting to binary incompatible change of (on)returnExceptional parameters 7273b139fff2 - this was an intentional and agreed on change (in spite of being against the compatibility policy)
- And at last, there was an API design problem - a return type of an API visible method was not public - e.g. visible
In all of these cases I manually updated the signature files to match the expected state. Then the compatibility check finally succeeded. Since today it will be way easier to keep BackwardCompatibility in our Truffle project!
Notification of Daily Changes
In addition to catching incompatible changes, it is useful to track compatible ones as well.
Some of the API changes happen by an accident and when they aren't noticed in time, they may become part of a release. But then they have to stay (as they are like stars) regardless how bad they are. Nobody wants to maintain erroneous code or API, so it is better to catch such changes as soon as possible and revert them before they propagate to a release.
Having a Hudson job to watch daily compatible changes and notify us about them is the best way to prevent accidental API changes. The job shall have two parts: generate new snapshot of today's APIs:
$ mx sigtest --generate $ zip snapshots.zip truffle/*/*sigtest
and make them persisted artifacts of the job. The other part is to take these artifacts from the last successful build and compare them against current state:
$ wget $HUDSON_URL/lastSuccessfulBuild/snaphots.zip $ unzip snapshots.zip $ mx sigtest --check all
then it is just a matter of sending the output of the last command to an observer mailing list, so humans can decide whether the change is intended or not.
I believe that with infrastructure like this the Truffle project will be able to stick to its promise of compatibility and better preserve investments done by language writers porting their languages to the fastest (J)VM on the Planet!