Incremental deployment

From APIDesign

Revision as of 18:23, 12 August 2011

When talking about the need for BackwardCompatibility I often talk in context of distributed development. Without some sort of BackwardCompatibility you can't expect independent groups to develop on top of each other's libraries. This might imply, that BackwardCompatibility is only important when multiple (distributed) parties are involved. Paradoxically That would be a false conclusion. Hear the following story!

NetBeans 7.0.1 Bugfix Release

NetBeans 7.0 was released in May 2011. Since then we worked hard on subsequent bugfixing release, to be out in August 2011. Users of NetBeans bugfix releases have two options: either download whole bits from scratch or update their existing installation in place using plugin manager.

Traditionally we have problems upgrading (as that is not scenario tested during development) and bugs are discovered only by our quality department just at the time of release (maybe we could tight up our processes a bit). The release 7.0.1 was not an exception. There was a severe bug in the update scenario. But this one was very curious.

Update only needed pieces

The update process as used by NetBeans these days has hidden gotchas. First of all NetBeans has always been a modular system. However over time we had to mitigate various problems, so now we have a system that is not only modular, but also an example of incremental deployment.

At the time of release 6.1 we realized that although we are offering various downloads (like Java edition, C/C++ edition, PHP edition, etc.), about 80% of people are downloading the full edition. Most of them never used all the technologies, but when the price is good (I mean NetBeans is for free) why not take it all? I knew why, because it would be bigger, slower and contain more bugs. However this was not what our customers realized, thus we had to do something with it. We invented Features on Demand! As a result people pay only for what they use even if they download everything.

Two releases later we realized that Features on Demand! can be broken by updates of new modules. Such modules, often by default turned on, turn on also pieces of the Features on Demand! functionality effectively rendering it useless and in some cases even broken. To mitigate this problem we decided to updates only that part of functionality that is enabled. That is good solution, as it lowers requirements on network bandwidth (only pieces that you really use are downloaded) and it also solves all previously known problems.

This was the state of the system at the time of NetBeans 7.0 release - modular and fully incremental deployment and enablement of all modules.

An incompatible change

During bugfixing we felt one of the bugs is most easily fixable by doing an incompatible change to one of our API libraries. The library had a restricted list of users, so we knew that just two modules depended on it. Morever all three modules were maintained by the same guy, thus we faced no problem during compilation. It was easy to make all the necessary changes at once, during one Mercurial changeset. The change was properly marked as being incompatible, so the NetBeans Runtime Container knew whether the classes will or will not link. So far so good.

Now combine this incompatible change with the incremental deployment system we had in place in release 7.0! Under some circumstances users of NetBeans 7.0 could be using the library and just one module depending on it. That is why, when we offered update to release 7.0.1, only those two would be upgraded. However, then the user might have a need to enable other installed functionality (comming from NetBeans 7.0) including the other module depending on the incompatibly changed library.

That did not work, as there was one module requesting library in old version, another module requesting library in new version and only one of these library versions could be active. The system collapsed and users were forced to delete it and reinstall from scratch!

Easy fix

For a moment this looked like unsolvable situation. We could fix the deployment to be less incremental. But that would require changes to the overall infrastructure, with the risk of being more disruptive than helpful. We could return back to 6.9 state (e.g. update even disabled modules), but we knew it had it own problems. We could try to load two incompatible version of the same library into the same JVM and hope for the best. But we had just one day to fix the problem!

Obviously (or surprisingly from certain point of view), it turned out that the simplest fix is to eliminate incompatible changes and solve the problem in BackwardCompatible way. At the end it was not that hard. Rather than adding new methods into interface, or changing signatures of existing methods, it was enough to create a new sub-interface (as described in ExtendingInterfaces).

This is not terribly nice solution, but who cares!? Millions of users are waiting for new version of NetBeans and are we arguing whether one guy can change three modules/source files in compatible way? C'mon, look at the problems from global perspective - just sacrifice yourself (as a writer of an API), the rest (e.g. million minus yourself) will benefit.

Being compatible vs. Update it all!

Most of the books, most of the methodologies deal with systems under complete control of some gate keeper/system administrator. Many people assume that these are the only systems to deal with. My experience comes from maintaining a framework and I can confess that we never have control over (almost) anything.

We used to believe we control API that we don't expose to public, but the latest adventures with incremental deployment show that even such assumption is over estimated. In truly distributed system you have little control and the only thing to rely on is (self) compatibility.

TBD.