API Design – API Evolution & API Versioning

Many people talk about API versioning – but versioning is actually only the mitigation of an issue that arises from software evolution. Sometimes we may not even need new API versions, because we can handle API evolution in a much smoother way. In this article, I give you both aspects — avoiding versioning by evolution management, and several means of actually implementing the versioning in a RESTful API.

Successful software always gets changed.
— Frederick P. Brooks

The Evolution Challenge

Managing change in software systems is never easy, but it is especially difficult to manage change in loosely-coupled distributed systems, such as API solutions. In loosely-coupled distributed systems not only the software components themselves are distributed, but also the responsibilities of the different components are distributed over different organizations, companies and people.

The API provider has little control over the client implementations of the consumer that use the API. The API provider might not even know all applications of the consumer, which call the API. This means that an unknown number of unidentified software components might rely on the API.

Already a small change in the API is enough to break some of the clients consuming the API. From the perspective of the API consumer, longevity and stability are important aspects of published APIs. The simple rule is, that the externally observable behavior of an API (from the perspective of the clients) cannot be changed, once the API has been published.

If the interface of the API changes anyway, it is impossible for the API provider to change all the apps consuming the API, and it is equally impractical to force all consumers to adapt or update their apps. API consumers are typically not willing and not interested in dealing with APIs that change frequently and will quickly abandon APIs that force them to rewrite their apps.

The Simple Rule for API Evolution

The simple rule for API evolution is, that the externally observable behavior of an API (from the perspective of the clients) cannot be changed, once the API has been published. Already a small change of the API might break some of the clients consuming the API. It is impossible to update all the consumers or at least unrealistic since they are under control of different owners. Thus, longevity and stability are important aspects of published APIs.

The restriction imposed by this rule might sound severe and even counter-intuitive since APIs are often developed using an agile development approach. Agile approaches are based on feedback loops and the idea of many incremental changes in the software. The agile development approach still applies to new or unpublished APIs.

Before the publication of the API, any change can be implemented in an agile manner. As soon as the API is published, however, the game changes. When APIs are published, they become available for consumers and it has to be assumed that the consumers build apps relying on the APIs. Published APIs cannot be changed in an agile manner. At least, APIs need to stay backward- (and forward-) compatible, so that old clients do not break and new clients can use the new and improved features.

Publication: The Root Cause of the Evolution Challenge

The root cause of the evolution challenge is the publication of the API. Once the API has been published, any externally observable behavior of the API cannot be changed without breaking clients.

The restriction imposed by this rule might sound severe and even counter-intuitive since APIs are often developed using an agile development approach. Agile approaches are based on feedback loops and the idea of many incremental changes in the software. The agile development approach still applies to new or unpublished APIs.

However, as soon as APIs are published, the game changes. When APIs are published, they become available for consumers and it has to be assumed that the consumers build apps with the APIs. Published APIs cannot be changed in an agile manner. At least, APIs need to stay backward (and forward) compatible, so that old clients do not break and new clients can use the new and improved features.

Types of API Evolution

People may want to change various aspects of published APIs. Are all of these changes equally severe for the clients? In this section, we analyze potential changes and classify them according to their severity. Severe changes are those changes that are incompatible and break a client. Not so severe are those API changes, that do not impact the client. They are called backward compatible.

Backward Compatible Changes

An API is backward compatible if an unchanged client can interact with a changed API. The unchanged client should be able to use all the functionality that was offered by the old API.

If a change is supposed to be backward compatible, certain changes to the API are prohibited, others are possible. The following is a list of backward-compatible changes:

• Adding query parameters (they should always be optional).

• Adding header or form parameters, as long as they are optional.

• Adding new fields in JSON or XML data structures, as long as they are optional.

• Adding endpoints, e.g. a new REST resource.

• Adding operations to an existing endpoint, e.g. when using SOAP.

• Adding optional fields to the request interfaces.

• Changing mandatory fields to optional fields in an existing API.

Incompatible Changes

If a change to the API breaks the client, the change was incompatible. In general, removing and changing aspects of the API leads to incompatibilities. A non-exhaustive list of incompatible changes:

• Removing or changing data structures, i.e. by changing, removing, or redefining fields in the data structure.

• Removing fields from the request or response (as opposed to making it optional).

• Changing a previously optional request field in the body or parameter into a mandatory field.

• Changing a previously required response field in the body or parameter into an optional field.

• Changing the URI of the API, such as hostname, port or path.

• Changing the structure or relationship between request or response fields, e.g. making an existing field a child of some other field.

• Adding a new mandatory field to the data structure.

Incompatible changes should be avoided if possible. If the change needs to be made, a new, additional version of the API has to be created, which exists in parallel to the existing API version.

Conclusion of the Analysis

Compatible changes can be implemented with an in-place update, the client does not break and no new version needs to be created. Incompatible changes would break the client and thus need to be offered as a new version. A recommendation is to perform only non-breaking, i.e. additive changes.

Anticipating and Avoiding Evolution

Since evolution is difficult to manage, APIs should ideally be built in such a manner, that evolution becomes practically unnecessary and that any foreseeable changes can be realized as compatible changes. This is, however, not always possible, because changes cannot always be anticipated.

The next best approach to anticipating changes is a mature development process. A mature development process may be able to avoid evolution to some extent. The need for evolution can be reduced by eliminating some of the most common causes of evolution. Evolution becomes necessary if the API has been published prematurely. From a business perspective, there are forces to publish as early as possible. From a long term support perspective, there are forces to publish as late as possible, to avoid evolution cycles. Good results can be achieved by compromising between both forces. This can be done by working closely with 1-2 pilot consumers of the API.

API Versioning – Coping with Evolution

If evolution cannot be avoided by anticipating or avoiding changes we need to cope with the change. For this purpose, we need to study the change more closely: Is it a compatible or an incompatible change?

Compatible changes do not break the clients. They should be reflected by a new minor version, which is fully compatible with the previous version.

Incompatible changes to published interfaces will break all clients. Incompatible changes need to be implemented in such a way that existing clients are not affected. This can be accomplished by creating a new major version of the API and maintaining the unchanged API alongside the changed API. Creating and publishing a new version of the API is overhead and thus slows down any changes to the API. Moreover, major versions need to be visible to the consumers and they need to be accessible by the consumers. There are different techniques for making major versions accessible, as discussed in the following sections.

Realize API Versioning in Accept Header

The client can use the HTTP AcceptAccept header to explicitly indicate the version of the API. The Accept header still contains the MIME-types as usual, but in addition, the version is appended. The URI of the API does not contain any version information. For new versions, the URL does not change, but the request header.

Accept: application/json; version=1

It is recommended not to introduce a default version. Without any default version, clients are required to always specify the version they expect and can deal with. This type of mandatory versioning requires that versioning is introduced right from the start when the first version of the API is published.

This versioning concept in the Accept header may be extended further. The extension offers the possibility to version two aspects of the API separately: the API (version 5 in the example) and the JSON schema (version 3 in the example):

Accept: com.domain.v3.application/json; version=5

Realize API Versioning as URI Path Parameter

The most common technique for versioning uses a URI parameter with the version number. For example:

https://domain.com/v1/API

Advantages of this approach are the widespread use and browser compatibility. Different versions can be explored via the browser.

Realize API Versioning in a Custom HTTP Header

A custom HTTP header could be defined for the version. For example X-Version: 1.1

In general, the use of custom HTTP headers is not recommended. This approach can lead to problems with caching. This approach is not recommended.

Realize API Versioning as Query Parameter

A query parameter could be defined for the version. For example:

https://domain.com/API?version=1

This approach is not recommended.

Realize API Versioning as a new Subdomain

A subdomain could be defined for the new version. For example:

https://v1.domain.com/API

This approach is not recommended.

HATEOAS API Versioning via Links

Hypermedia solves the versioning issue by indirection via a link resource. A link resource is comparable to a pointer in the C programming language or a symbolic link in the file system. A link resource has a separate identity and a separate URI from the versioned resource that it points to. The resource that it points to still needs to be versioned using e.g. the URI Path. But the link resource is not versioned. The link resource is updated to always point to the latest version of the versioned resource. The only address that is communicated to the client is the URI of the link resource, whose address never changes. In this way, the HATEOAS style can provide a solution to the evolution problem.

How can linking be implemented? Well, there are several options. For example via an HTTP redirect (301 or 302) with the URI of the versioned API in the HTTP LocationLocation header. Another option is including the URI of the versioned resource into the payload of the link resource.

Supporting Multiple Versions Simultaneously

You have versioned your API and you are ready to share the new major version with the world? Great! But what to do with the old version?

You may keep the old version of the API running, at least for a certain grace period. Notify your consumers and convince them of the benefits of the new version. Engaged consumers will eventually switch to the new version.

Keeping the old version of the API running is another option, but a costly one: infrastructure, monitoring, support, and documentation have to be provided for all active versions. Providing several versions may also confuse new API consumers looking for documentation. So eventually old versions should be retired and switched off. You will have to live with the fact, that some consumers will not make the switch to the new version of the API.

Want to read more on Versioning or REST in general?

This is an excerpt from the RESTful API Design Book.