Distributed development

From APIDesign

Revision as of 12:04, 18 December 2017

Applications are no longer written from scratch. They are assembled from modular pieces coming from elsewhere.

It is relatively easy to consume well written libraries developed by completely independent teams (think of JUnit, Spring, Guava, etc.). Developers of these components are relatively skilled to keep BackwardCompatibility - a first per-requisite for distributed development.

On the other side of the system there are the end developers consuming the libraries. Their behavior is important as well. However the good thing is that the users of such libraries are fine to consume them as immutable component - e.g. as binaries. With tools like Maven it is then easy to assemble your application in a modular way. Distributed development seems to work quite fine in this setup.

Up stream library developers work on completely isolated schedule and down stream end developers have the freedom to upgrade to newer components at their own will. Things tend to work fine, until...

Bugfix Upstream

From time to time there is a need to fix something in the up-stream component. What can one do then? In a well organized teamwork, one would submit a patch (well an API Patch as it has to change an observable aspect of the library, e.g. an API) to the upstream, used local fork of the project temporarily, wait for the upstream project to produce fixed version and switch to it once it is published. That would be ideal state, but...

... it is too much hassle! Rather than that the end developers hack. They workaround issues in upstream projects and once they get their own application running, they forget about the workaround. That is indeed the simplest solution at the moment - why think about future development? Why be afraid that a workaround may stop working in subsequent versions? Why worry now?

However nasty, this kind of development seems to work fine. Modularity of really distributed and distinct teams works!

Inside one Team

On the other hand, when one tries to apply the same principles to teams inside the same organization, one starts to get into troubles. The closer the teams are, the less acceptable it seems to be to consume just binaries and wait weeks (or at least days) for new version to be produced. Also, the closer the teams are, the less they think about BackwardCompatibility - suddenly the whole distributed development idea starts to be unsound.

Distant Teams

For some reason the art of organizing the teamwork for distributed development or at least adjusting one's development style in such away is easier for really independent teams. Once you have a separated, but connected team (like I have in case of Truffle, Graal and various GraalVM language teams), everything starts to look impossible.

Suddenly people tend to require upstream fixes as soon as possible. They want to access sources of such components and modify them without any restrictions. They stop thinking about the project as being composed from modular components - rather they want to work with all of them as a whole. That is obviously more comfortable, but also it is not scalable.

Looks like everyone is worshiping modularity, but nobody wants to give an inch up to really have it.