I recently had a discussion with a client about the evolution of the API program over the last 15 years. As I shared the various changes I had seen, I began to realise that today’s API programs look much different today, including the mixture of web-based APIs, event-based APIs, and now data-oriented APIs. In this three part series, I want to share my observations with you and help you to navigate the changing landscape of the API program. Whether you are embarking on a new API journey or at some point along the way, you will find both empathy for the challenges you are facing and some guidance on how to handle the new complexity of your API program.
The complex API portfolio
The majority of APIs within an organisation’s portfolio are web-based and often fall into the classification of REST-based APIs. Web APIs provide interfaces to data and behaviour to deliver digital capabilities, typically over HTTP. These web-based APIs were meant to handle course-grained system interactions between web and mobile apps, partner integrations, and internal system integrations.
If you are further into your API journey, you may even have an Open API Specification (OAS), RAML, or API Blueprint description file for each API. You may even have a process for which you organise these design-based artefacts to ease the burden of API consumers attempting to discover and integrate with your various APIs.
Yet, today’s API portfolios are becoming more heterogeneous. To address today’s more sophisticated integration and automation scenarios, it is no longer enough just to assume that there is one style of API to be designed and documented. We are now seeing a mixture of API styles and protocols that are being combined to address today’s more complex API needs.
These expanded set of APIs styles and protocols offer a variety of granularity and interactions to address today’s needs, including:
- Business capability APIs
- Experience APIs
- Event-based APIs
- Service APIs
- Stream and data-oriented APIs
- Serverless and Edge APIs
- Intent-based APIs for infrastructure and network automation
In this series, we will look at each one of these API styles that are becoming part of our API portfolio.
API Style 1: Business capability APIs
Many organisations I consult to primarily build APIs that deliver data. Their web APIs apply the CRUD pattern: create, read, update, and delete. While this is often necessary as a starting point, API programs that purely expose databases via CRUD APIs become anaemic; they lack the depth and robustness necessary to compose the more complex solutions today’s organisations typically encounter. Some organisations are realising that CRUD APIs are insufficient to meet today’s needs and are embarking on the higher-value business capability APIs that focus on the needs of their customers, partners, and workforce first.
Below is an example of a typical web-based API that supports account management:
Business capability APIs have a few distinct traits to support the more robust interactions:
They are operation-oriented, offering queries (fetch/filter) and/or commands (transactional) as the basic element of interaction. These operations are combined to realise the outcome(s) desired by the end users.
Additionally, transactional boundaries are enforced for each operation, to avoid the need for transaction managers to coordinate distributed transactions in the past, such as those used with SOAP and WS-Transaction. When longer running transactions are required, API resources are designed to capture intermediate state of the transaction, usually with a series of operations that perform add/remove/modify operations on the transaction via API operations.
They are more course-grained than a single database table, to optimise the user interaction within a bounded context. Business capability APIs are often associated with aggregates in domain-driven design (DDD) terminology. This prevents exposing internal implementation details, such as domain models, or wrapping database tables, including link tables, as CRUD operations.
They also contain all necessary business rules to avoid pushing business logic to the client by ensuring that operations enforce rules within the API rather than forcing it upon the front-end developer. This doesn’t mean the user interface can’t have robust display behaviour, but it does mean that all user interfaces behave consistently because they do not need to re-implement the same business logic for web, mobile, voice, and integration scripts.
Finally, business capability APIs seek to optimise client interactions by supporting client-side and intermediary caching, reducing network round-trips. These optimisations may also be used for concurrency control through the use of conditional HTTP requests to avoid overwriting data changes from other API requests.
Each capability is composed of one or more operations, sometimes referred to as endpoints. These operations combine a unique URL with an HTTP verb. API gateways, such as Tyk, are designed to manage business capability APIs built as web-based APIs.
Business capability APIs may be used directly from web and mobile apps. However, there has been an increase in the use of experience APIs as well.
API Style 2: Experience APIs
Business capability APIs are composed of one or more endpoints that are often combined into a sequence of request/response interactions to deliver the capability desired. However, this interaction may be sub-optimal. Experience APIs seek to solve this problem.
Experience APIs, sometimes referred to by its pattern name of backend-for-frontend (BFF), orchestrate multiple business capability APIs into a single API endpoint. These styles of APIs solve a number of problems beyond business capability APIs:
- Optimising APIs calls over slow or unreliable networks, such as from mobile devices or for apps meant to operate in limited conditions
- Offering device-specific API endpoints, such as those used to support voice-based interaction models
- Providing a more efficient front-end developer experience, through the use of query-based API styles such as GraphQL
While REST-based APIs have been the choice for some time for experience APIs, I have seen an increase in the use of GraphQL to support front-end APIs that power web and mobile apps. GraphQL shifts the traditional HTTP-based API approach of an endpoint-per-resource to a single endpoint that supports a query language to request the desired data by a client app.
GraphQL APIs are organised in terms of types and fields, rather than endpoints common with business capability APIs using REST. GraphQL uses a query-based API style to allow clients to define the breadth and depth of the data desired in a single request.
Below is an example of the previous REST-based API, using a GraphQL query:
More complex GraphQL queries may be used to create experience APIs, mapping the various data types supported in a single query to multiple business capability API requests on the backend. Additionally, GraphQL supports the concept of mutations for modifying data or executing commands. However, organisations have had varying degrees of success with mutation support, so the most common use of GraphQL that I’ve seen is for augmenting existing REST-based APIs to add complex query support and for orchestrating multiple backend capabilities into a single request.
It is important to note that API gateways are able to manage and protect GraphQL-based APIs but only at the single URL where queries are submitted. It is up to the GraphQL server to apply fine-grained access controls itself, or defer this to the API gateway through custom plugins written to enforce the specific GraphQL queries supported based on the API consumer’s permissions.
API Style 3: Event-based APIs
Event-based APIs support surfacing business events outside of a department or the entire organisation over HTTP. New solutions are built on top of these business events, making it easier to integrate systems without introducing tight coupling.
While most organisations use the publish/subscribe pattern to decouple different systems through message-oriented middleware (MOM) such as message brokers, this approach has been limited to integrations within the organisation. Now, event-based APIs are emerging as a way to extend this pub/sub model over HTTP, enabling integration to be extended to partners and customers as well.
I’m seeing growth in a number of technologies, such as WebSockets and Webhooks, added to an organisation’s API portfolio to offer event-based APIs. Event-based APIs eliminate the use of continual polling of REST-based APIs to check for data changes. Instead, events are emitted over HTTP to notify interested parties about important changes. This shifts the integration of our APIs from polling to event-driven extensions that support emerging use cases beyond what our APIs were originally intended to support.
AsyncAPI is a standard for capturing the description of these events across different protocols. It was designed with a syntax similar to OpenAPI Specification, making it easy to author. Below is an example of an AsyncAPI specification for account events that our account-based APIs, shown in part 1 of this series, may publish:
In this example, we see an Account Created event that may be used by other systems to trigger deeper API integration, or simply to keep two or more systems in sync.
Event-based APIs enable your internal teams, partners, and customers to react to internal changes or important events while reducing the burden of a polling-based approach that increases the load on your infrastructure. Event-based APIs are the next big opportunity for APIs, as I’ve written before. API design that neglects the offering of event-based APIs will likely experience reduced growth as partner and customer demands go beyond existing business capability API offerings.
Since event-based APIs use a combination of callbacks and/or push notification, managing and protecting event-based APIs may require a combination of an API gateway and other techniques to prevent unauthorised access to your events.
Part 1 Conclusion
So far, our API portfolio is comprised of business capability APIs that digitise the organisation’s core capabilities, along with experience APIs that help map these capabilities to device or use-case specific scenarios. We also have looked at how event-based APIs allow our API portfolio to be extended beyond its original use cases. But how does our API portfolio evolve when we consider things such as microservices and data streaming? We will examine these types of APIs in part 2 of this three-part series.
