ContinuousIntegration

From APIDesign

There is nothing wrong on having your own ContinuousIntegration server like Jenkins or Hudson set up. In fact having it can only help quality of your project. As Chapter 9 of TheAPIBook explains, Chapter 9 is the sole of good frameworks and the most important part of good projects.

Can't Test Everything!

On the other hand, people get so used to this comfortable safety net provided by the by ContinuousIntegration systems that they tend to forget there is also something behind the horizon. QA departments are well aware of the fact that regardless of the amount of CodeCoverage (btw. how much CodeCoverage is enough?), there can always be errors visible to end users. The situation is even more complicated when it comes to API Design.

A common advice would be to increase your CodeCoverage or add integration tests to verify the real workflow. The biggest problem when designing an API of a framework however is: you can't run all the users code! You don't have it, you don't even know about it. When your framework is successful, then running all projects that use your API wouldn't even scale.

Observe the Visible

As such one has to do something else. Rather than testing all the dependent code, one has to make sure the externally observable aspects of your framework as stable - e.g. they don't share like the amoeba in the Amoeba Model. What does that mean?