API Design – Prototyping

Plan to throw one away; you will anyhow.

–Frederick P. Brooks

Prototyping is a preparation phase for productive implementation. One goal of prototyping is to learn as many practical insights as possible while spending as little effort as possible. This can be achieved by quickly creating a simple prototype implementation, that you plan to throw away. Not every aspect of the API should be implemented in this phase, but only the critical aspects of the API are assessed. Assessment of the prototype is performed from a technical perspective by the engineers and from a usability perspective by pilot consumers. Typical design decisions that are explored during prototyping are the backend design decisions and the non-functional properties of the system.

There are thus two goals for proper prototyping: practical insights into critical implementation and usability issues and a low effort for the creation of the prototype.

The first goal is to gain practical insights through the prototyping effort. To gain some learning with practical relevance, the API prototype needs to be as realistic as possible. While simulations can be considered to be low-fidelity prototypes, this phase creates high-fidelity prototypes that are more realistic, more relevant, and closer to the actual implementation. The API Prototype should conform to the API description and use real data from real backends.

At the same time, there is the second goal, which requires the prototype to be built as quickly as possible and with as little effort and budget as possible. To achieve the necessary speed, the implementation does not have to be pretty, does not have to be optimized, and may contain engineering shortcuts.

To fulfill both goals, code generation can be used. Code generation for API proxies is offered for all API description languages. Properly generated code conforms to the API description. However, the generated code is merely the “bones” of a skeleton, only the interface of the API can be generated. The generated code skeleton provides some structure and the correct interface, but the “meat”, the actual implementation, has to be added manually around the skeleton. The missing code can be added with relatively low effort since the skeleton already provides a structure.

So which implementation tasks need to be done during this phase? This needs to be decided on a case-by-case basis. If the real backends are available, they may be integrated, otherwise, a simulation of the backend is used. Requests and responses of the backends need to be transformed, input and output need to be validated and security needs to be implemented and configured, just to name a few. Some implementation details can be left out at this stage, such as traffic shaping or performance optimizations.

An API prototype is always an imperfect and incomplete implementation of the API. Actually, the prototype implementation has to be incomplete, otherwise, too much time has been scheduled for realizing the prototype. To maximize the learnings from prototyping, one should focus on implementing the aspects, which are most critical. For one API, the backend connection may be on the critical path, for another API, it may be a complex input validation algorithm. Focus on these critical issues and use shortcuts for the other issues to get to a testable prototype quickly. If the backend connection is not on the critical path, the prototype API does need to be connected to the real backend and a simulation of the backend is sufficient at this stage.

For simple APIs without any critical issues or the need to learn anything before implementation, one might be able to hop over the prototyping phase and go directly to the implementation phase.

Validation: Acceptance Tests with Pilot Consumers

API prototypes are usually built to answer the question “What are the major hurdles for building this API?” Besides exploring the feasibility, the prototype can be used for acceptance tests by pilot consumers. Let’ s see what this means.

In general, an acceptance test is a black-box testing method, where users test if the specifications and requirements of a system are met. Acceptance tests are used to verify the completeness of a system. In our case, API consumers test the API prototype. Ideally, they use the API when designing or building their app. In an acceptance test of the API, the consumers answer the question “Does this API provide some value for my app?”

Pilot consumers need to be API consumers, who are willing to work with unfinished APIs with changing interfaces, broken clients, frequent updates, unavailability, and low performance of the API. In short: a pilot customer must be able to bear some pain. This is why pilot consumers are typically recruited from inside the organization of the API provider, for example from a department of the API provider. In some environments, pilot consumers are also called beta testers. Ideally, the pilot consumer writes an app that solves a real problem, sometimes a pilot consumer may just write a demo app for testing the API.

Why would an API provider voluntarily become a pilot consumer? The advantage for pilot consumers is early access to innovative APIs, allowing for a short time-to-market of the consumer’s app. This is an advantage that should not be underestimated in an ecosystem, where time-to-market has a high impact on the market share.

Besides the validation by pilot consumers, the API prototype should also be checked for conformance with the API descriptions. To some extent, conformance between implementation and API description is already ensured by code generation. However, generated code may have been changed and manually added code may still need to be checked. Such a test should include JSON well-formedness checks and JSON schema validation of the results. Both of these tests can be generated from the API description. Additional tests might need to be added manually.