Building a sound API testing strategy

When building an API platform, it is important to have an API testing strategy. Selecting the right approach for API testing is key to the success of the programme’s supportability and enabling faster development in the future.

What do we hope to achieve through API testing? How will we reach these goals? What value are we ultimately providing by this testing? These are the questions that this article will answer.

What will we achieve through API testing?

Before we discuss how to test an API service, we need to make note of our goals. To that end, we need to ensure that our APIs are tested for:

  • Correctness
  • Reliability
  • Performance and Scalability

Correctness

First and foremost, API testing needs to ensure the correctness of the API’s operation. Handling thousands of clients per minute does not do us any good if the information that the API is providing or acting on does not meet the API’s specification. Squashing bugs, hunting out inconsistencies, and verifying that an API meets the spec against which it has been designed all fall under the umbrella of testing for correctness.

Reliability

After moving past verification of correctness, API testing next focuses on reliability. Is the API providing the same correct information every time it is accessed? Do the same actions produce the same results? Is the API ever unreachable? These questions must be answered to ensure a business value can be derived from the API.

Performance and Scalability

Only after correctness and reliability of an API have been ascertained can we turn our focus to performance and scalability testing. Will an API meet the needs for the number of the customers we expect it to service? How much will we need to scale the service? Will it scale horizontally? These are the challenges that performance testing will help bring to light.

How we will achieve our API testing goals?

To reach our API testing goals, we will break down our testing into five areas, using the API Testing Pyramid:

  • Unit testing
  • Functional testing
  • Operational monitoring
  • Security protection
  • Acceptance testing

Unit testing

Unit testing is the first and smallest section of the API Testing Pyramid. Unit tests are the fastest tests to implement and execute, but also provide limited value to business stakeholders. These tests answer the question, “Is each code module working properly?” Unit tests also make up a portion of our performance and scalability testing, albeit at the micro level.

Unit tests can be used to prevent internal regression of bugs by isolating portions of the API implementation. Unit testing is most useful in areas of code than cannot easily be intercepted from the outside, such as code using third party services (logging, mail), and is also important in security-conscious areas, such as authentication, authorisation, or encryption.

Unit testing traditionally consists of code written in the same language as the service, and tests functions and methods that are internal to the API. Such tests are usually easy to automate, depending on language or framework, and can be tested locally without the need to deploy.

Functional testing

Functional testing, sometimes referred to as contract testing, is used to verify that each API endpoint meets the expected behavior and honors the API’s defined contract for the consumer. Functional testing answers the question, “Is each endpoint working to the specification?” and encompasses the bulk of our correctness testing.

API contracts are typically defined prior to implementation, and are verified as part of the functional testing process. Some common contract specification formats are OpenAPI (Swagger), API Blueprint, and RAML, but loosely-defined developer documentation approaches are also common. Contracts should be stable to prevent developers from need to constantly update their API consumer code, though some have been known to be less so than others.

Functional testing verifies all of the following from a specification:

  • Are input parameters being followed? How are bad inputs handled?
  • Are the expected outputs received?
  • Is response formatting correct? Are the proper data types used?
  • Are errors being handled correctly? Are they reported back to the consumer?

Like unit testing, functional testing is often automated. Automating these tests has a higher infrastructure cost, due to the need for additional common software components, such as a data store, but is still common in Continuous Integration/Continuous Deployment models today.

In more recent developments, new tools have emerged to programmatically verify APIs against OpenAPI specifications. Two such tools are Stoplight’s Prism and Apiary’s Dredd – the latter also includes support for API Blueprint. When combined with automated functional testing, these tools present a formidable foe against regressions sneaking into your API.

Operational monitoring

Why is it important to monitor APIs? APIs can and often do provide the primary interface for applications to interact with a system. By playing the role of a dependency, it is critical for the API service to be available, whether that is to other services which are internal to an organisation, or to paying customers. Additionally, there may be service level agreements (SLAs) that the company has agreed to, and could yield a negative financial result if missed, in addition to the prospect of angry or upset customers. Operational monitoring is a key component of our reliability testing goal.

There are a number of ways to verify continual availability of an API:

  • Third-party API monitoring-as-a-service is available from a range of companies, and often start free for a small number of services and becomes paid as host counts increases.
  • Open source monitoring tools are available that can be run internal to one’s infrastructure.
  • Custom tools built to perform load and performance testing can be modified to run less frequently and at a smaller scale for the purpose of monitoring or soak testing.

Analytics are also a key component to API operational monitoring. Analytics will verify that real-world usage matches what was seen in testing for both correctness and performance. Analytics measurements can be as simple as logging of performance counters, or as complex as integrating third-party libraries with extensive dashboard support.

Security protection

There isn’t a week that goes by with a company being hacked and private information exposed, or an Internet of Things botnet being used to take down a popular service. Security testing aims to answer the question, “Does the API protect against attacks and PII leaks?” Security is a process, not a product, and a continual one at that. Even with the ever-changing security landscape, we can still lay down an evolving set of best practices, and test against them:

  • Expect to be a target. Don’t believe that being a small company or hiding in a small network address space will keep you safe from prying eyes. Most attacks are automated and are constantly scanning for targets.
  • Whitelist, don’t blacklist. When setting access controls on resources, whether it’s at the network, operating system, or application level, take a default-deny stance. Only explicitly enable access as needed. In the case of an API, a user should have no access to resources by default, until explicitly granted it.
  • Encrypt all data in transit. TLS is the new normal, and services like Let’s Encrypt make it both free and trivial to implement on API endpoints. There is no excuse for unencrypted web traffic today.
  • Authenticate, then authorise. Authentication is the process of identifying a valid user. Authorisation is deciding what resources that user will have access to. An API will save time and computing resources by not performing expensive actions, such as database lookups or calls to third-party services, until the user has verified who they claim to be.
  • Stay up-to-date. To reiterate, security is a continual process. Operating systems, networking hardware, and applications all need to be constantly kept up to date to prevent successful attacks. Scanning tools are available at all three of these levels to ensure compliance and protection against known vulnerabilities.

Acceptance testing

Acceptance testing, also called solution-oriented testing, ensures that the API as a whole supports the intended use cases that it was designed to solve. Does the API solve real problems that our customers have? Does it do something that people actually care about? Whereas functional testing answers “Does each endpoint work?”, acceptance testing verifies the combination of endpoints to achieve a desired business outcome. Acceptance testing is the most valuable testing for an API, encompassing both correctness and reliability goals, and is where the most development time should be spent when under time constraints.

Similar to functional testing, acceptance testing is usually performed using black box testing methods. The user testing the system knows only the public interfaces of the system, in our case the API contract, and has no knowledge of the system internals. The tester uses only that API contract to verify that the system meets all expected end-to-end functionality. The internals of the API can and likely will change over the course of development, but this should not affect the results of the acceptance tests.

There are a wide range of tools available to aid in acceptance testing, both open source and proprietary, that support or can easily be adapted to API testing:

Tying it all together

By deciding on our testing goals and forming an approach to reaching them through the API Testing Pyramid, we have laid the foundation of a solid API testing strategy. Our testing will help to ensure API correctness, reliability, and performance/scalability. And our approach will be repeatable through a combination of layered testing approaches and selection of automation tools.

For more insight on API testing strategy take a look at our Tips for testing your API. This will be useful when it comes to building an API, a good test strategy ensures that your API both works correctly and meets the promises of its definition.

© Tyk Technologies, 2019