APIFest08

From APIDesign

As part of promotion of TheAPIBook and also the celebration of 10 years of NetBeans releases, we decided to play again the game described in Chapter 17, Using Games to Improve API Design Skills. The participants included primarily members of Czech JUG extended with few worldwide participants who wanted to play and wished to celebrate with all those who contributed to the NetBeans architecture, and design practices which make NetBeans platform the most stable Java rich client application framework.

Thanks to the courtesy of Arseniy Kuznetsov, director of NetBeans, the winners of the APIFest08 received a copy of TheAPIBook. Read more about the results.

There will be few rounds of the competition, during which the participants are supposed to solve tasks given to them in timely manner and send their results to the API Fest 08 mailing list. The whole competition will use rules like those of API Fest One.

Task 1

The first task consists of downloading the template. The preferred way is to use Mercurial and clone the apifest08 repository:

hg clone http://source.apidesign.org/hg/apifest08/

You will get an Ant project in the currency directory. It is designed to be opened in NetBeans IDE easily, but it can also be manipulated from command line or other tools by those who prefer pain over comfort.

Two sources files are important:

• currency/src/org/apidesign/apifest08/currency/Convertor.java - will contain the API that you should create. Feel free to introduce other classes, but please keep them within the org.apidesign.apifest08.currency package.
• currency/test/org/apidesign/apifest08/test/Task1Test.java - this is the class with actual tasks that your API has to fullfil. Please stuff the method bodies with code, so they demonstrate use of your API.

The project is currently not compilable (you need to make the Convertor class public somehow), compile it, compile its tests and execute the tests so they pass without a failure.

Remember that Runtime Aspects of APIs are as important as API signatures.

Finally zip your project and send it to the API Fest 08 mailing list by Friday, Sep 26, 2008.

cd apifest08/currency
ant clean
zip -r $HOME/myname-task1.zip .

We'll discuss your design decisions on Czech JUG session on Monday, Sep 29, 2008 together with announcement of the second task!

For those who will not make the Mercurial work for them, there is an alternative option: You can download source ZIP file from the repository snapshot, unzip it and use it directly.

As processing of the results might take some time, we need to set the following rules:

• members of CZJUG takes precedence
• first come, first served
• if we run out of our capacity, we can stop processing further solutions

Task one and half

Thanks you all, dear twelve brave men for participating in APIFest08! I was really glad to see your submissions after the task 1. I really like the variety between your individual solutions.

However, I also have to apologize. I did try, to correctly specify the task 1 goals, however, looking at some of your solutions, I failed. Some of your APIs do not satisfy all the use cases that I had in my mind. That is my fault, I should have expressed myself clearly. Anyway it puts us into bad situation. Now you are not at the same starting position for the real APIFest08 game. That is why I have to, like a starter seeing a problem before 100m run, call off the ready state, and ask you to prepare once more. That is why I am calling round one and half. Please adjust your your solution according to additional tasks, described in Task1Test.java. There is proper description of the behaviour of the factory methods and two more tests. Please implement this in your solutions.

Please do it by Tue, Sep 30, 2008, noon. You do not need to retain any backward compatibility yet, however restrain yourself from doing too big changes to your APIs, they are all good, just satisfy the additional use cases, please.

Please accept my apology, and good luck!

Task 2

The task2 of APIFest08 is here. We have fourteen solutions that advanced to second round. Some of them did use their chance to get up to speed in APIFest08:Task1.5, some of them did not. The latter ones will be in tougher position, as the second round is just about to start.

You task #2 is to allow one convertor to hold multiple rates of multiple currencies. Also, create an API to allow composition of convertors. Here is the Task2Test.java file with detailed description of the quest. Locate your solution among existing task2 directory, copy the Task2Test.java file along your Task1Test.java. Enhance your API, show its usage in the Task2Test.java.

After you are done, please generate a diff. The best way is to use Mercurial:

# create the repository
hg clone http://source.apidesign.org/hg/apifest08/
# or update it
hg pull; hg update
# copy the task 2 file
hg copy currency/test/org/apidesign/apifest08/test/Task2Test.java task2/solutionXY/test/org/apidesign/apifest08/test/Task2Test.java
# then do your modifications
# ...
# and finally produce a diff
hg diff

you can obtain the diff in other ways, potentially you can send us to our mailing list the whole ZIP file, but we want to have the right to process diffs first, as that is simpler for us.

Please submit your solutions by Sat, Oct 4, 2008. Good luck, and do not forget to keep BackwardCompatibility!

Task 3

The task3 of APIFest08 is here to tease you more, dear participants. Btw. great work on the previous tasks! It is really hard for me to come up with a task that will be unimplementable for you. I guess you all defend BackwardCompatibility of your solutions pretty well.

Let see what will happen during the round3. I have prepared the Task3Test.java for you. Please enhance your API to allow your API users to write online convertor that updates its exchange rates all the time. Please find the details of your quest inside of the Task3Test.java file.

Locate your solution among existing task3 directory. There shall already be Task3Test.java file along your Task1Test.java and Task2Test.java. Enhance your API, show its usage in the Task3Test.java.

After you are done, please generate a diff. The best way is to use Mercurial:

# create the repository
hg clone http://source.apidesign.org/hg/apifest08/
# or update it
hg pull; hg update
# then do your modifications
# ...
# do not forget to add newly created files
hg add task3/solutionXY
# and finally produce a diff
hg diff

you can obtain the diff in other ways, potentially you can send us to our mailing list the whole ZIP file, but diffs are preferred.

Please submit your solutions by Fri, Oct 10, 2008, morning of CET, so we can process your submission before weekend and think about more sophisticated ways to tease you. Good luck!

Task 4

It is unbelievable! We are approaching task4 of APIFest08 and we still have eight participants who play with us! Moreover the competition is still not decided, as far as I can tell. Today I spent few hours looking for BackwardCompatibility problems in provided solutions and I still have not managed to break three of them! It is not that surprising as some of the competitors follow advices of TheAPIBook. For example I've seen APIDesignPatterns:ResponseReply being used. Guys, how can I find a quest for you if you know all the good tricks? Anyway this means we have to go on, as I have only two books to give to winners!

That is why let me ask call round4: Please integrate with dates. Keep history of conversions, allow queries in historical data, etc. Please see Task4Test.java with the actual quests.

Locate your solution among existing task4 directory. There shall already be Task4Test.java file along your Task1Test.java, Task2Test.java and Task3Test.java. Enhance your API, show its usage in the Task4Test.java. Remove the conditional if failing statements.

After you are done, please generate a diff. The best way is to use Mercurial:

# create the repository
hg clone http://source.apidesign.org/hg/apifest08/
# or update it
hg pull; hg update
# then do your modifications
# ...
# do not forget to add newly created files
hg add task4/solutionXY
# and finally produce a diff
hg diff

you can obtain the diff in other ways, potentially you can send us to our mailing list the whole ZIP file, but diffs are preferred.

Please submit your solutions by Thu, Oct 16, 2008, morning of CET. I guess this is the last coding round. On Saturday you will start hacking solutions provided by others!

Judgment Round

The judgment day is here! Dear APIFest08 participants, during the last rounds, you were concentrated on making your own solution perfect. However since today you will have to stop looking just at your own solution, you will need to start digging in solutions provided by others to find possible evolution mistakes. This is a new task, different to what you have done so far, but I hope you'll make it. Because only if you find mistakes in solutions provided by others you will be allowed to win TheAPIBook.

If your own solution is found backward compatible and it successfully passed the 4th round, you'll be awarded with 10 points. By default you all have those points. However you can be awarded with 1 point by showing compatibility problems in each solution provided by someone else, who advanced to 4th round. Of course, in case somebody finds such problem, the author of the solution looses own 10 points. As I wrote in the last post, I know that at least five solutions have some problems, so do not give up and seek for them! Your points are waiting! Also please note that the fact that I may know about a problem of your solutions does not mean other participants are going to find it. Don't give up and good luck!

First of all you need to setup your working environment:

# create the repository
hg clone http://source.apidesign.org/hg/apifest08/
# or update it
hg pull; hg update
# create directory named by you Joe Hacker would have:
mkdir taskx/jhacker

and now your hunt can start. Repeat following tasks as many times you wish. First of all somehow select the solution you want to find problem in. Usually I use a diff:

diff -ru task2/solution03 task3/solution03

Each difference is potential candidate for compatibility problem. It is just necessary to write code against the old API that does something different in the new version. If I notice some potential evolution problem, then I setup the against project and write a test into it to show the error:

# select the solution you want to break, let supposed it is #3:
hg copy taskx/jtulach/against-solutionXY/ taskx/jhacker/against-solution03
# now open some of the created files in your favorite editor:
netbeans --open taskx/jhacker/against-solution03/project.properties

Now adjust the content of the properties file:

# change definition of apitotest property to solution03:
apitotest=solution03
# you can also adjust the taskA and taskB variables to 
# reflect the days that you want to compete against
# for example if there is compatibility problem between
# task2 and task3, change them to:
taskA=${apifest}/task2
taskB=${apifest}/task3

Now you can open the project in the NetBeans IDE

netbeans --open taskx/jhacker/against-solution03/

or directly edit its jhacker/against-solution03/test/apifest/CurrencyTest.java file. The goal is to make this file compilable against api available in the version taskA. If you execute the project:

ant -f taskx/jhacker/against-solution03/build.xml run

or by pressing F6 in the NetBeans IDE, your test is compiled, executed against taskA and it has to succeed. If it does, the test is executed against taskB library. If it fails, you have demonstrated BackwardCompatibility problem and that is why the build succeeds. Make sure you open the Output Window to see the real resul! There is also the JUnit Result Window and it is not really helpful in the against project, but I do not know how to suppress its appearance. If the test succeeds with taskB library, the sources of your test are compiled once again against taskB library. This can fail, if there are source incompatibilities. Failure means your success, as you have demonstrated an incompatibility! To see result of these steps in action, look at my attack against solution02 (I can show it as it did not advanced into this final task and you cannot be awarded with points by finding problems in it). If you managed to attack one solution, perfect, you can repeat these steps for another one.

The rules for writing compatibility tests are similar to API Fest One - e.g. the less tricks you use the better. Where trick means more or less Java Permission. Do not try to use reflection that could obviously break each solution. Also do not use wildcard imports in source, that can easily break each solution as well. Do not put the code into the API package, that would allow you to use package private methods and would be unfair. Rules are simple: less tricks is better and I am the judge. Enjoy!

After you are done, please generate the diff using hg diff and submit your solutions by Sun, Oct 26, 2008, morning of CET to our mailing list. Possibly you can ZIP your sources and send them to us too. As soon as we process your submissions, I'll write an APIFest08 report and announce winners. Also I will prepare TheAPIBook for the winner.

Final Report

Would you win?

The celebration of 10 years of NetBeans releases is in progress and that is why it is also time to celebrate all those who contributed to the NetBeans architecture, and design practices which make NetBeans platform the most stable Java rich client application framework.

The best way to have fun is to play games. The best way to learn is to play as well. The best way to teach is to organize such game. When me and CZJUG lead Jakub Podlešák realized that, we decided to organize similar game as described in Chapter 17 of TheAPIBook. It is a game to teach, learn and have fun while designing architecture and APIs and practising BackwardCompatibility principles in software interfaces.

There were four base rounds (1, 2, 3, 4) in the APIFest08. We started with fourteen participants. However as the competition advanced further and further, only those who were really eager to win managed to finish all four design tasks. Still, six solutions advanced to the final judgment day/week.

The judgment round is in fact the start of the real competition. However it is also much more difficult than the first four rounds. Instead of looking into own solution, one needs to dig into solutions provided by others and seek for left compatibility problems which is different and time consuming. Still I am glad that two participants, Petr Šmíd and Jan Žák did not give up and sent me their hacks showing BackwardCompatibility evolution problems in many solutions.

Problems in Solution 04

Both guys managed to find some problem in solution 04. Actually there were multiple problems as demonstrated by Petr, however the most obvious one was source incompatibility caused by the addition of new method into a subclassable interface as shown by Jan and me (my solutions are just illustrative, I am not one of the competitors). The lesson to take is that if one is seeking for 100% BackwardCompatibility one needs to prevent modifications to classes that can be subclassed.

Problems in Solution 06

The author of solution six did few compatibility mistakes. First of all, as Jan demonstrates, the once public field Convertor.one disappeared in subsequent revision, which is clearly incompatible for clients using it.

I found an interesting source incompatibility caused by adding overloaded version of constructor. If there are two methods with the same name and number of parameters, but different types, one can try to call them with null and the compiler will refuse the compilation:

against-solution06/test/apifest/CurrencyTest.java:18: 
reference to Convertor is ambiguous, both method
  Convertor(BigDecimal,java.util.Currency,java.util.Currency) 
and method
  Convertor(Convertor.RateProvider,java.util.Currency,java.util.Currency)
match
        Convertor c = new Convertor(null, cur1, cur2);

Obviously the code compiled before the second constructor was introduced. It failed with later version. This is not that big incompatibility in practise, but it is enough to loose in APIFest08.

Petr exploited another functional incompatibility. The text of a thrown exception changed between two releases. As such it is enough to generate the exception and compare the text. A thing to remember? There are different kinds of APIs, text messages being one of them. For purpose of APIFest08 all of them are equal.

Problems in Solution 07

When I checked the state of all solutions after task 3, I found a perfect way to win. The solution 07 is an example of absolute API stability. The author of this solution managed to guess my future requirements and he changed nothing in the original API in first three rounds! Indeed, it is impossible to find an inconsistency in two versions of an API if they are identical. That is why my own goal for task 4 was to force the author of solution 07 to make some changes. I succeeded, the API version 4 is different than the previous versions, however as the solution 07 is based on a variation of request/response pattern, it still seems unbreakable.

And actually nobody managed to question BackwardCompatibility of the solution. But Jan Žák noticed an important problem. To quote his email: Sometimes - like in the Task2 - the author even added nothing to the API and implemented the requested feature directly in the test case. I'm not convinced, that this approach is strictly fair.. When I look at the quest of the second round and see the complex subclassing classing code provided by solution 07...

/** Merge all currency rates of convertor 1 with convertor 2.
* Implement this using your API, preferably this method just delegates
* into some API method which does the actual work, without requiring
* API clients to code anything complex.
*/
public static Convertor merge( final Convertor one, final Convertor two ) {
  // quite a complex code with subclassing
}

...I have to agree. All other solutions got this correctly. As such I am giving Jan one point for finding a problem in solution 07 - finding that it does not meet the requirements.

Problems in Solution 11

This is the solution designed by Jan Žák and it seemed completely unbreakable for me. However, Petr had other opinion. He realized that when a code like:

private String k1, k2;
public Constructor(String k1, String k2) {
  this.k1 = k1; this.k2 = k2;
}
public boolean canConvert(String s1, String s2) {
  if (k1.equals(s1) && k2.equals(s2)) {
    return true;
  }
  if (k2.equals(s1) && k1.equals(s2)) {
    return true;
  }
  return false;
}

is modified to be more general:

private List<String> keys = new ArrayList<String>();
public Constructor(String k1, String k2) {
  keys.add(k1); keys.add(k2);
}
public boolean canConvert(String s1, String s2) {
  if (keys.contains(s1) && keys.contains(s2)) {
    return true;
  }
  return false;
}

The result is not the same. Petr managed to exploit this with his test. Congratulation, this is very clever!

This shows how carefully we need to evolve APIs of our libraries. Even slight change in Runtime Aspects of APIs can lead to incompatibilities.

Problems in Solution 13

This solution suffered with source as well functional compatibility problems. Jan managed to exploit both, Petr concentrated on the source incompatibility and I on the functional one.

The thing to remember is that as soon as you implement equals, you need to stick with its implementation. Changing it can be exploited either by careful hackers or by everyone who puts the objects into List or other collection classes.

Problems in Solution 14

This is the solution provided by Petr Šmíd. As Petr is one of the last standing hackers, his solution is not easy to break. I managed to exploit change in implementation classes by using obj.getClass().getInterfaces() check. This does not require any additional permissions, everyone can do this, however it is not really fair. If this was the only way to break Petr's solution, I would still consider his work unbreakable.

However Jan Žák created more intrinsic exploit that generates ArrayStoreException, if executed with newer version of the API. The reason for that is that the task4 version wraps all CurrencyRate with own TimeLimitedCurrencyRate objects. As these objects are not used only internally, but also exposed to clients with a getter getCurrencyRates, it is enough to pass in own objects and check that the returned values are of the same type. Excellent work Jan!

The lesson to remember is to Separate APIs for Clients and Providers as advocated in Chapter 8.

Results

All solutions are flawed in one or other way. As such nobody gets ten points for creating an unbreakable solution. The only points awarded are only for exploits:

Petr Šmíd

Petr is one of the winners of APIFest08.

Petr broke solutions 4, 6, 11, 13.

Jan Žák

Jan broke solutions 4, 6, 13, 14 and challenged correctness of solution 7

Celebration

Both Petr and Jan visited in Sun's Prague office on Thursday 30, 2008 and got their copy of TheAPIBook. Jan is the absolute winner of APIFest08 - he broke four solutions, plus correctly challenged another one. However, I'd like to congratulate you all, dear participants. I enjoyed the game and I hope you are not sorry for participating either.

Thanks, Jaroslav Tulach!

<comments/>

Lessons Learned

What worked and what did not?

• Round 1.5 was necessary and it is good we planned it in advance. It should be part of any future fest.
• The points shall be assigned in a way to stimulate the final judgment - e.g. it should be possible to win even if one's solution is found broken
• solution07 was violating requirements, however players of the API fest were not ready to report that as a problem. This shall be improved next time.

That is all, happy API designing!