When building an API platform, it is important to have an API testing strategy. Selecting the right approach for API testing is key to the success of the programme’s supportability and enabling faster development in the future.
What do we hope to achieve through API testing? How will we reach these goals? What value are we ultimately providing by this testing? These are the questions that this article will answer.
Why do you need API testing?
There are several reasons why we do API testing. API testing helps ensure APIs function and perform as they should. It helps us understand any limitations in terms of load and identify any security vulnerabilities. Ultimately, API testing contributes to more robust and reliable API products.
Different types of API testing
You can choose the API testing methodology that best meets your API testing purpose, thanks to the wide range of different tests available. Your API performance testing strategy needs to consider both what you need to achieve and how you plan to do so.
Whether you’re starting with the basics of how to test an API or simply refining your API test strategy, the testing types below should help ensure your APIs are the best they can be.
Validation testing
With any new API, you’re likely to be eager to validate correctness as soon as you’ve designed the logic. You don’t need a fully functioning user interface (UI) or app to do so; you can run an API unit test or functional testing at this early stage.
For those new to functional testing vs unit testing, the former is about testing the API as a whole, while the latter is about testing components individually.
Functional testing
Any test plan for API testing should ensure your API functions as expected. Functional testing checks this, looking at your API’s code level-functionality. It ensures everything works as defined in your documentation. Functional testing covers areas such as ensuring endpoints are accessible, how the API handles valid and invalid values, whether requests and responses behave as expected and much more.
Security testing
Any API testing approach should include security testing. This ensures your authentication and authorisation mechanisms function as expected, checks access control, scans for vulnerabilities to injection attacks and checks data is secure in transit.
Load testing
Load testing reveals how your API performs under differing loads, including peek loads. Combined with scalability and stress testing, this enables you to understand what you can expect from your API in performance terms as usage grows.
Runtime and error detection
Including runtime and error detection in your API performance testing strategy means you can focus on issues with monitoring, execution, resource leaks or error detection. All important components of API testing.
Penetration testing
Wondering how to test API defences? Penetration testing can help, examining how your API holds up against a range of threats.
Fuzz testing
Fuzz testing examines what happens when your API is overloaded with a huge volume of random data, to help you prepare for worst case scenarios.
UI testing
UI testing can provide deeper insights into the overall experience of using the API, making it a worthy addition to any API test plan.
What will we achieve through API testing?
Before discussing how to test an API service, it’s time to make note of our goals. To that end, we need to ensure APIs are tested for:
- Correctness
- Reliability
- Performance and Scalability
Correctness
First and foremost, API testing needs to ensure the correctness of the API’s operation. Handling thousands of clients per minute doesn’t do any good if the information that the API is providing or acting on doesn’t meet the API’s specification. Squashing bugs, hunting out inconsistencies, and verifying that an API meets the spec against which it has been designed all fall under the umbrella of testing for correctness.
Reliability
API testing next focuses on reliability. Is the API providing the same correct information every time it is accessed? Do the same actions produce the same results? Is the API ever unreachable? These questions must be answered to ensure a business value can be derived from the API.
Performance and Scalability
Next comes performance and scalability testing. Will an API meet the needs for the number of the customers we expect it to service? How much will we need to scale the service? Will it scale horizontally? These are the challenges that performance testing will help bring to light.
How we will achieve our API testing goals?
To reach our API testing goals, we will break down our testing into five areas, using the API Testing Pyramid:
- Unit testing
- Functional testing
- Operational monitoring
- Security protection
- Acceptance testing
Unit testing
Unit testing is the first and smallest section of the API Testing Pyramid. Unit tests are the fastest tests to implement and execute, but also provide limited value to business stakeholders. These tests answer the question, “Is each code module working properly?” Unit tests also make up a portion of our performance and scalability testing, albeit at the micro level.
Unit tests can be used to prevent internal regression of bugs by isolating portions of the API implementation. Unit testing is most useful in areas of code than cannot easily be intercepted from the outside, such as code using third party services (logging, mail), and is also important in security-conscious areas, such as authentication, authorisation, or encryption.
Unit testing traditionally consists of code written in the same language as the service, and tests functions and methods that are internal to the API. Such tests are usually easy to automate, depending on language or framework, and can be tested locally without the need to deploy.
Functional testing
Functional testing, sometimes referred to as contract testing, is used to verify that each API endpoint meets the expected behavior and honors the API’s defined contract for the consumer. Functional testing answers the question, “Is each endpoint working to the specification?” and encompasses the bulk of our correctness testing.
API contracts are typically defined prior to implementation, and are verified as part of the functional testing process. Some common contract specification formats are OpenAPI (Swagger), API Blueprint, and RAML, but loosely-defined developer documentation approaches are also common. Contracts should be stable to prevent developers from need to constantly update their API consumer code, though some have been known to be less so than others.
Functional testing verifies all of the following from a specification:
- Are input parameters being followed? How are bad inputs handled?
- Are the expected outputs received?
- Is response formatting correct? Are the proper data types used?
- Are errors being handled correctly? Are they reported back to the consumer?
Like unit testing, functional testing is often automated. Automating these tests has a higher infrastructure cost, due to the need for additional common software components, such as a data store, but is still common in Continuous Integration/Continuous Deployment models today.
In more recent developments, new tools have emerged to programmatically verify APIs against OpenAPI specifications. Two such tools are Stoplight’s Prism and Apiary’s Dredd – the latter also includes support for API Blueprint. When combined with automated functional testing, these tools present a formidable foe against regressions sneaking into your API.
Operational monitoring
Why is it important to monitor APIs? APIs can and often do provide the primary interface for applications to interact with a system. By playing the role of a dependency, it is critical for the API service to be available, whether that is to other services which are internal to an organisation, or to paying customers. Additionally, there may be service level agreements (SLAs) that the company has agreed to, and could yield a negative financial result if missed, in addition to the prospect of angry or upset customers. Operational monitoring is a key component of our reliability testing goal.
There are a number of ways to verify continual availability of an API:
- Third-party API monitoring-as-a-service is available from a range of companies, and often start free for a small number of services and becomes paid as host counts increases.
- Open source monitoring tools are available that can be run internal to one’s infrastructure.
- Custom tools built to perform load and performance testing can be modified to run less frequently and at a smaller scale for the purpose of monitoring or soak testing.
Analytics are also a key component to API operational monitoring. Analytics will verify that real-world usage matches what was seen in testing for both correctness and performance. Analytics measurements can be as simple as logging of performance counters, or as complex as integrating third-party libraries with extensive dashboard support.
Security protection
There isn’t a week that goes by with a company being hacked and private information exposed, or an Internet of Things botnet being used to take down a popular service. Security testing aims to answer the question, “Does the API protect against attacks and PII leaks?” Security is a process, not a product, and a continual one at that. Even with the ever-changing security landscape, we can still lay down an evolving set of best practices, and test against them:
- Expect to be a target. Don’t believe that being a small company or hiding in a small network address space will keep you safe from prying eyes. Most attacks are automated and are constantly scanning for targets.
- Whitelist, don’t blacklist. When setting access controls on resources, whether it’s at the network, operating system, or application level, take a default-deny stance. Only explicitly enable access as needed. In the case of an API, a user should have no access to resources by default, until explicitly granted it.
- Encrypt all data in transit. TLS is the new normal, and services like Let’s Encrypt make it both free and trivial to implement on API endpoints. There is no excuse for unencrypted web traffic today.
- Authenticate, then authorise. Authentication is the process of identifying a valid user. Authorisation is deciding what resources that user will have access to. An API will save time and computing resources by not performing expensive actions, such as database lookups or calls to third-party services, until the user has verified who they claim to be.
- Stay up-to-date. To reiterate, security is a continual process. Operating systems, networking hardware, and applications all need to be constantly kept up to date to prevent successful attacks. Scanning tools are available at all three of these levels to ensure compliance and protection against known vulnerabilities.
Acceptance testing
Acceptance testing, also called solution-oriented testing, ensures that the API as a whole supports the intended use cases that it was designed to solve. Does the API solve real problems that our customers have? Does it do something that people actually care about? Whereas functional testing answers “Does each endpoint work?”, acceptance testing verifies the combination of endpoints to achieve a desired business outcome. Acceptance testing is the most valuable testing for an API, encompassing both correctness and reliability goals, and is where the most development time should be spent when under time constraints.
Similar to functional testing, acceptance testing is usually performed using black box testing methods. The user testing the system knows only the public interfaces of the system, in our case the API contract, and has no knowledge of the system internals. The tester uses only that API contract to verify that the system meets all expected end-to-end functionality. The internals of the API can and likely will change over the course of development, but this should not affect the results of the acceptance tests.
There are a wide range of tools available to aid in acceptance testing, both open source and proprietary, that support or can easily be adapted to API testing:
Building an effective API testing strategy
An effective API testing strategy supports the creation of more reliable API products. Include all of the following in your strategy for best results.
Creating a dedicated testing environment
This enables you to test thoroughly and catch and address any issues before you deploy to your production environment.
Automating API tests
Use automation to save time, standardise your API testing approach and minimise the potential for human error.
Running tests throughout the API lifecycle
Test as early as possible to head off later issues, then implement a continual testing approach to ensure nothing breaks whenever you make changes to your API or any connected components.
Writing reusable subtests
Save time and effort and enhance consistency by writing reusable subtests where possible.
Designing for scalability
Keep things organised when creating your testing framework to help you avoid wasted time and duplicated test logic as you scale and as your API evolves.
Tying it all together
Deciding on testing goals and forming an approach to reaching them through the API Testing Pyramid lays the foundation of a solid API testing strategy. Testing will help ensure API correctness, reliability and performance/scalability, with repeatability achieved through a combination of layered testing approaches and automation tools. Remember: a good test strategy ensures that your API both works correctly and meets the promises of its definition.
For more insight on API testing strategy read these tips for testing your API.
