How to run an efficient APIM evaluation

APIs are important, and that’s why managing them is important.

Choosing the right platform to manage them is essential; there are many options available, and there is no “objectively the best” platform; which is suitable for your use case, your team, or your organisation will depend on a range of factors. The ideal choice may change over time, as what’s right today may not be right tomorrow.

Making that assessment will be costly in three ways:

  1. $ – every hour spent on the assessment costs cash money. An evaluation which utilises one day a week, of five people, over four months is an 800-hour investment, or USD 45,737, on an average $115K software engineer salary.
  2. Opportunity cost/pace – every hour spent on assessment is spent on something other than your business capability. A six-month evaluation versus a two month means four months of delay to market, etc.
  3. Making the wrong choice – this is the worst cost – not only do you take on costs one and two, but also the cost of building on the platform for a year, only to find it doesn’t deliver what you need and need to re-platform. This can cost you millions.

And, more technically speaking:

  • Reputational cost of bad choices with consumers, partners or even internal stakeholders.
  • Governance costs of bad decisions, causing regulatory issues or mismatched alignment with internal workflows.
  • Lack of API adoption due to misalignment with consumer needs.
  • Incompatibility with industry standards.

So here is our guide to making a good assessment, controlling costs, and avoiding mistakes.

TL;dr: it needs some prep, rigour in setting scope and process and a close partnership with vendors. Luckily we’ve laid it all out below, along with examples and guidelines based on years of experience and the best customer satisfaction scores in the industry.


First, you need requirements – they should map to business requirements. If you don’t have them, or they’re just a checklist of features, stop. Go get them or talk to our sales team to help establish your API management requirements.

We’re talking about having a collection of API-related issues that, when optimised, will result in more dollars for your business, either by reducing expenses or increasing revenue.

Remember – Your API Management requirements should map to business requirements rather than technical ones. We’ll address the technical needs later.

Here are some examples:

Time-boxing the evaluation

Usually, your business requirements and KPIs are tied to a date. At Tyk, we suggest setting aside two weeks for your evaluation. I know – it seems short for essential plumbing. How is this possible?

1. Forget About Infrastructure

Anyone can visit and run a free five-week Enterprise trial on our fully Managed service, Tyk Cloud. They can spin up a Tyk environment and, in 10 minutes, be testing their business requirements.

We want users to focus on API Management, not the management of API management.

You shouldn’t be architecting how a product fits into your infrastructure during the PoC. A PoC is to see if a product will solve your business problems. Stop worrying about technology for a second.  

No one has ever said “no” to Tyk because it can’t fit into their organisation’s deployment preferences.  Ever. However, do you need a drop-in replacement for Istio retry policies? Or you’re looking to upgrade your Federation v2 spec’d supergraphs for whichever reasons?   

Yes, save weeks of engineering time by starting with THAT.

The bottom line is: Your infrastructure is not that unique.

What is unique are your business requirements.

It’s a question if the APIM platform you’re evaluating can support your weird legacy auth that will take 10 months to deprecate whilst you move users onto a shiny new identity provider.

It’s less of a question about whether or not it can run behind your DMZ on your vSphere or OpenShift cluster (Tyk is the most flexible product on the market!).

Whether you’re on a private cloud, public cloud, Kubernetes, Linux, Windows, Legacy green screen, a Raspberry Pi, an IoT device, on a truck, on a cruise ship, you name it, we’ve seen it.

Yes, you may want the sign-off of your lead DevOps engineer before agreeing to make the purchase. And you want to know how the platform will fit in your CICD pipelines.

These are things that can be answered downstream of or in parallel to the sales process. Ideally, they’d happen during the dedicated joint-implementation phase alongside one of Tyk’s customer engineers, but we understand this takes a tremendous amount of vendor trust, which we’ll talk about later.

2. Control scope

When making a large investment, it’s normal to want the sign-off of multiple stakeholders. However, the risk introduced by increasing your stakeholder surface area is increasing scope. You may find very suddenly that the CRO is demanding adding a PAYG API monetisation strategy on top of your initial requirements after hearing about the various APIM features.  

This is going to throw a wrench in your plans to expose APIs to a vendor if you’re not able to defend the scope.

Stay focused – prove the initial value, or minimum viable project, before adding an additional scope. Remember, everything you do to extend the evaluation also extends the go-to-market time and risks your project never surpassing an increasingly extensive list of sign-offs.

3. Shift right on benchmarking

Performance benchmarking. Every company thinks they need to do it as part of their evaluation.  For every option.

Imagine you’re purchasing a fleet of Trucks. This is an essential part of your logistics chain. It’s important that they’re:

  • Reliable, not going to break down on the road
  • Strong, capable of carrying your load, 
  • Scalable, easy to scale alongside your company’s growth, 
  • Comfortable, how comfortable the cab is for your employees, etc. 

You know what’s NOT on this list? The 0-60 speed of trucks.

It’s doubtful that you would choose a fleet because the truck does zero to sixty 0.2 seconds faster.

Let’s get down to brass tacks; it’s to Tyk’s detriment to advise you this way; Tyk has the most performant Gateway on the market. We write all our software in Golang, the language of choice for performance-critical applications such as Kubernetes and Docker.

The bottom line: If performance was ever given as a reason NOT to purchase an API gateway product, then that product needs to go the way of the dodo.

What if hyper-scale is critical to your use case? You’re an exception. You, folks, should absolutely make this investment in engineering hours, as performance IS business requirements. Nanoseconds matter.

For everyone else, save your money, and save your time. Or at least invest in this exercise AFTER completing your business requirements.

4. Treat your vendors like partners

Buying software, especially plumbing like API management, is difficult. Unfortunately, this means that customers typically extend the PoC period as long as possible to reduce risk to alleviate the anxiety of whether or not they’re making the right strategic move.

This behaviour wouldn’t be necessary if a vendor relationship could become a partner relationship. This is why choosing a partner, not a vendor, is essential.

At Tyk, we recognise this and have built a solutions architect team that exists solely to help users map their business requirements into architecture and implementation plans which fit their infrastructure.

This team’s only job is to bridge a customer’s world and ours. You’re advised to leverage their experience and expertise and consider them an extension of your teams. You don’t have to do this alone.

Comparing build vs buy

We’ve worked with many users over the years who decided to roll their product, invest tremendous resources into this path and then quickly out-scale it. Then they speak to our solutions architects about migration paths and become happy Tyk customers. Yes, there’s some survivorship bias baked in there, but the anecdata is legit.

If you’re determined rolling your own is the right solution, you can build on top of existing open source gateways like Tyk’s or start from scratch, using spring API Gateway to write your libraries if you’re a Java shop.

Before you do that, please evaluate both options’ total cost of ownership (ToC). Some companies build their own because they forecast it as the cheaper option.

Here is my word of warning to those who choose to build over buy for something complex like APIM: Technology moves quickly, making it difficult to keep systems up-to-date. Especially if the system needs maintenance and is not your primary engineering focus. APIM has become a fast-growing industry precisely because APIM is a complex field.

With that said, make sure you factor in:

  • Cost of engineering hours
  • Cost of product maintenance + support
  • Cost of infrastructure complexity increase
  • Cost of security / Pen testing
  • Cost of documentation and training.

If you’re Netflix and have extremely custom requirements for an API gateway, it makes perfect sense to debate this. However, the other 99 percent of companies are being negligent; wasting their time and money to allow what is an otherwise fun engineering project like this to happen.

Making a good assessment

Choosing the right API management platform is crucial for your business, but making the assessment can be costly in terms of time, opportunity cost, and potential for making the wrong choice. 

To make a good assessment, consider the following steps: 

  1. Identify your business requirements
  2. Time-box the evaluation
  3. Focus on API management rather than the technology
  4. Control scope
  5. Shift right on benchmarking 
  6. Treat vendors as partners, and leverage the expertise of Solutions Architects.

Finally, when comparing building vs. buying an API management solution, consider the total cost of ownership and the challenges of maintaining a custom-built system in a fast-paced industry. 

By following these guidelines, you can make a more informed decision on the right API management platform for your business.