Designing an API? Good luck!

If you’ve developed software to some extent, you’ve probably used dozens if not hundreds of APIs, so called Application Programming Interfaces. In short, APIs are the visible part of a library or framework that you include into your project. In reality, the last sentence is a complete lie. Every programmer at some point got bitten by some obscure behavioural change in a library that wasn’t announced in the interface (or at least the change log of the project). There’s a lot more to developing and maintaining a proper API than keeping the interface signatures stable.

A book about API design

practicalapidesignA good book to start exploring the deeper meanings of API development is “Practical API Design” by Jaroslav Tulach, founder of the NetBeans project. Its subtitle is “Confessions of a Java Framework Architect” and it holds up to the content. There are quite some confessions to make if you develop relevant APIs for several years. In the book, a game is outlined to effectively teach API design. It’s called the API Design Fest and sounds like a lot of fun.

The API Design Fest

An API Design Fest consists of three phases:

  • In the first phase, all teams are assigned the same task. They have to develop a rather simple library with an usable API, but are informed that it will “evolve” in a way not quite clear in the future. The resulting code of this phase is kept and tagged for later inspection.
  • The second phase begins with the revelation of the additional use case for the library. Now the task is to include the new requirement into the existing API without breaking previous functionality. The resulting code is kept and tagged, too.
  • The third phase is the crucial one: The teams are provided with the results of all other teams and have to write a test that works with the implementation of the first phase, but breaks if linked to the implementation of the second phase, thus pointing out an API breach.

The team that manages to deliver an unbreakable implementation wins. Alternatively, points are assigned for every breach a team can come up with.

The event

This sounds like too much fun to pass it without trying it out. So, a few weeks ago, we held an API Design Fest at the Softwareschneiderei. The game mechanics require a prepared moderator that cannot participate and at least two teams to have something to break in the third phase. We tried to cram the whole event into one day of 8 hours, which proved to be quite exhausting.

In a short introduction to the fundamental principles of API design that can withstand requirement evolution, we summarized five rules to avoid the most common mistakes:

  •  No elegance: Most developers are obsessed with the concept of elegance. In API design, there is no such thing as beauty in the sense of elegance, only beauty in the sense of evolvability.
  •  API includes everything that an user depends on: Your API isn’t defined by you, it’s defined by your users. Everything they rely on is a fixed fact, if you like it or not. Be careful about leaky abstractions.
  •  Never expose more than you need to: Design your API for specific use cases. Support those use cases, but don’t bother to support anything else. Every additional item the user can get a hold on is essentially accidental complexity and will sabotage your evolution attempts.
  •  Make exposed classes final and their constructor private: That’s right. Lock your users out of your class hierarchies and implementations. They should only use the types you explicitly grant them.
  •  Extendable types cannot be enhanced: The danger of inheritance in API design is that you suddenly have to support the whole class contract instead of “only” the interface/protocol contract. Read about Liskov’s Substitution Principle if you need a hint why this is a major hindrance.

The introduction closed with the motto of the day: “Good judgement comes from experience. Experience comes from bad judgement.” The API Design Fest day was dedicated to bad judgement. Then, the first phase started.

The first phase

No team had problems to grasp the assignment or to find a feasible approach. But soon, eager discussions started as the team projected the breakability of their current design. It was very interesting to listen to their reasoning.

After two hours, the first phase ended with complete implementations of the simple use cases. All teams were confident to be prepared for the extensions that would happen now. But as soon as the moderator revealed the additional use cases for the API, they went quiet and anxious. Nobody saw this new requirement coming. That’s a very clever move by Jaroslav Tulach: The second assignment resembles a real world scenario in the very best manner. It’s a nightmare change for every serious implementation of the first phase.

The second phase

But the teams accepted the new assignment and went to work, expanding their implementation to their best effort. The discussions revolved around possible breaches with every attempt to change the code. The burden of an API already existing was palpable even for bystanders.

After another two hours of paranoid and frantic development, all teams had a second version of their implementation and we gathered for a retrospective.

The retrospective

In this discussion, all teams laid down arms and confessed that they had already broken their API with simple means and would accept defeat. So we called off the third phase and prolonged the discussion about our insights from the two phases. What a result: everybody was a winner that day, no losers!

Some of our insights were:

  • Users as opponents: While designing an API, you tend to think about your users as friends that you grant a wish (a valid use case). During the API Design Fest, the developers feared the other teams as “malicious” users and tried to anticipate their attack vectors. This led to the rejection of a lot of design choices simply because “they could exploit that”. To a certain degree, this attitude is probably healthy when designing a real API.
  • Enum is a dead end: Most teams used Java Enums in their implementation. Every team discovered in the second phase that Enums are a dead end in regard of design evolution. It’s probably a good choice to thin out their usage in an API context.
  • The most helpful concepts were interfaces and factories.
  • If some object needs to be handed over to the user, make it immutable.
  • Use all the modifiers! No, really. During the event, Java’s package protected modifier experienced a glorious revival. When designing an API, you need to know about all your possibilities to express restrictions.
  • Forbid everything: But despite your enhanced expressibility, it’s a safe bet to disable every use case you don’t want to support, if only to minimize the target area for other teams during an API Design Fest.

The result

The API Design Fest was a great way to learn about the most obvious problems and pitfalls of API design in the shortest possible manner. It was fun and exhausting, but also a great motivator to read the whole book (again). Thanks to Jaroslav Tulach for his great idea.

8 thoughts on “Designing an API? Good luck!

  1. Maybe it’s true in Java-Land but I have to disagree with some points you made.

    1. “No elegance”. You are locking yourself out of potential users then. Let’s take Python for example where I’d rather work with something beautiful such as Kenneth Reitz (read up his comments on designing APIs) requests library than the utter crap urllib is. And if done right, evolvability is orthogonal to beauty.

    2. “Extendable types cannot be enhanced”. This sounds nice if your API is for pure consumption but what if the user has to extend the API in order to achieve a certain goal? I cannot imagine a simple GUI widget API that is not extendable by sub-classing.

    • I participated in the experiment and feel like I can clarify the remarks.

      1. If you look at it from the user’s perspective then you certainly want an API that allows you to solve problems in an elegant way. But to enable such an elegant API, you have to pull some dirty tricks on the inside. So the “No elegance” part refers to the API developers point of view. The requirements for the API change over time and the assumptions you previously made get invalidated. You would usually refactor your code and the interfaces but in the API developer’s world this means breaking existing code. To counter this, you have to patch things up with nasty conditionals and switch statements. The team I was in encountered such a situation. I’d like to describe it but I don’t want to spoil the experiment for others.

      2. The GUI problem is pretty much solved and the requirements are well known and unlikely to change. If you make something where the requirements are less well known, you should forbid extensibility as much as possible and only allow it when you feel safe to do so. Adding a method to an interface at the bottom of your hierarchy will break the entire codebase.
      If subclassing is substantial for your API you must be absolutely sure you got the base types right. You can drive away users with breaking changes as quickly as you’ve won them with your extensibility.

      • Thanks for your comment on that. But …

        1. I don’t buy the that “to enable such an elegant API, you have to pull some dirty tricks on the inside.” – this sounds like unproven folklore to me. Yes, changing requirements (which is the worst of all evil) will certainly make writing elegant APIs difficult. But again, in my opinion beauty is orthogonal to evolvability and changing one does not necessarily mean changing the other. But I won’t argue here, because, it looks like we _will_ spoil the fun for others😉

        2. “If you make something where the requirements are less well known, you should forbid extensibility as much as possible and only allow it when you feel safe to do so.” – that’s exactly my point I wasn’t able to convey. I agree that careless extensibility is going to cause havoc, but dogmatic prohibitions won’t help either and can make writing client code awkward.

      • Thank you for your input. My explanation of the concepts is rather rude and to the point. For a more elaborate and trustworthy discussion of the point “beauty in API design”, I can recommend the book mentioned in the article.

        I can agree to the observation that elegance and backward compatibility can be orthogonal. A skilled API designer will probably produce such a beauty everytime. The book (and the workshop) refers to API design beginners (aka normal programmers). As a programmer, the concept of beauty is deeply engrained while the concept of backward compatibility doesn’t matter. With the war cry “no elegance”, I tried to reverse the value set. And, as a matter of fact, the API Design Fest doesn’t hand out points for beautiful solutions, only for backward compatible ones.

        I can understand that elegance is enjoyable and important for developers. As a “business-sided” developer, backward compatibility trumps elegance for me everytime the two aspects aren’t orthogonal (what might be considered a reasonably lack of skill of the API designer).

    • Many APIs are built on top of existing or growing standards, like HTTP or JSON. I think in those scenarios it’s rather unlikely that sudden and unpredictable requirement changes, like the one simulated in the experiment, will occur.
      Just look at the hardware specs (sensors, pixel density, IO ports etc.) of a new smartphone and then those of a two year old model. I don’t know if elegance and evolvability is still orthogonal in the hardware API world, but I feel safer to assume that it’s not.

      • I don’t get your example. What does a hardware spec has to do with an API?

        Let’s assume, you mean what I think you mean, and that means you want to access* a certain hardware device with an evolvable and elegant API. As a matter of fact, I happen to work with real hardware (scientific cameras to be precise) and provide a generalized API to the devices. As far as I can see, I managed to create a stable API that covers all use cases as well as any future hardware devices without ever having to change the API.

        * Whatever that means.

      • What I was thinking about were things like motion APIs where gyroscopes where added at a later point in time, or the iPhone aspect ratio that was kind of taken for granted and then changed. You have to implement a fallback solution for things like that (which was a fairly inelegant black bar in the case of the iPhone’s aspect ratio)

        I believe I’ve found the misunderstanding. I think you were talking about the overall beauty of an API and that it should be a feature to begin with (to which I agree) and I was talking about a beautiful API that gets an ugly dent because of unexpected requirement changes, which I still think can totally happen. In my scenario someone has to pay the price, either the API developers or the API users.

  2. *sigh* Why is backwards-compatibility always so much more important than elegance/correctness for you enterprise guys?*

    That’s just wrong! If it’s so expensive to fix your old code then maybe something is wrong with your code. New users of some library shouldn’t have to pay for the bad code quality of older users of the same library.

    Sure, use clean deprecation paths where possible and give library users ample time to react, but don’t let them stop you from improving your library. Because otherwise we’ll end up with fun things like urllib/urllib2 (how could zilluss not mention urllib2 — that it’s just as unusable as urllib is half of the fun!) or little nuggets like java.util.UUID.equals that *will* bite new users.

    * yes, I’ve worked on multi-million LOC projects too

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s