Performance-tuning your Tyk API Gateway

As a client-facing engineer at Tyk, I’m often asked “How fast is Tyk’s API Gateway?”. My answer is always: “damn fast – and feature packed too!”. But how do you get the optimum performance from your Tyk installation?

In this post we’ll be walking through step-by-step how you can get some insanely low latency and high throughput out of a tiny commodity server. We will get down to the nitty gritty and show you some actual numbers to help effectively tune and size your Tyk Gateway installation.

I will be showing you how to:

  • Increase throughput from ~5000 requests per second to ~6400 requests per second
  • Reduce 95th %ile gateway latency by ~54% (9.2ms to 4.2ms)
  • Reduce CPU consumption by ~40%

In order to achieve all of the above, we will be tweaking just a single environment variable: GOGC.

We will be deploying Tyk Gateway, Redis Database, Upstream Server & Load Generator ALL to a single commodity 2-core DigitalOcean Droplet.

In client production deployments, we have seen Tyk Gateways performing TLS termination, Authentication, Rate-Limiting & Quota checking plus analytics recording alongside Statsd instrumentation – with the Tyk Gateway introducing just 1.5ms of latency at a sustained mission critical 60k transactions per second.

This performance blog is not supposed to be scientifically accurate – if you want to do that, feel free to re-create in a more sanitised environment and let me know your results over on the Tyk Community.

Does that sound like fun? Let’s get benching!

But first, how not to deploy Tyk

When you view showcase WordPress themes they are almost always gorilla-slow: that is, very large and with assets that are megabytes in size when the browser downloads. In real life, you will never use every single asset that the theme provides, you will likely minify and apply a few tweaks, only enabling what you need for the production site.

With this in mind, I’ve seen a couple of benchmarks of Tyk in the wild, and it is instantly clear that the deployment option chosen was a quick-start installation. While this is perfect for a few key use-cases (see next paragraph), it’s important to remember that, much like a WordPress theme, the easiest & fastest way to deploy Tyk is never going to be optimally configured.

The installation I refer to is the Tyk Pro Docker Demo, which is an all-in-one installation of the professional edition of Tyk. It is great at showing off features & functionality of Tyk as an enterprise API Management Platform, which can be deployed pretty much anywhere docker is available, but the amount of resource contention involved when benchmarking against this docker-compose micro-service architecture should not be surprising.

By design, Tyk is deployed as a collection of cloud-native micro-services, all working harmoniously together. Possibly the only micro-services architecture API management solution in the wild to manage your micro-services. As such, taking an out-of-the-box docker-compose to benchmark it without taking the time to understand how each component interacts with each other will almost certainly result in sub-optimal and inconsistent results.

Sizing & Configuration

This short section will provide some insights on how to go about sizing Tyk & its dependencies.

Tyk’s API Gateway is possibly the lightest and most performant solution in the industry, its only dependency being a small Redis Server.

That’s right – there is no need for MongoDB, Cassandra, Postgres or any of these heavy, computationally & memory expensive databases for an Open Source installation of Tyk.  In AWS, you could simply use ElastiCache, if GCP, you could use MemoryStore.

Redis

Redis requires enough ram to hold your tokens, temporary analytics and fast networking.

A default Redis installation will persist to disk in a non-optimal fashion, in doing so, is likely to choke the Gateway with the default configurations. For simplicity, we would recommend disabling AOF to reduce disk space usage. SAVE locks the database, and as such, BGSAVE will allow snapshotting in a non-blocking fashion.

In a real-world scenario, we recommend a Redis-cluster deployment, which will allow for linear performance improvements by sharding the data which is especially useful when you enable authentication, handle rate-limits and quotas. With Redis Cluster, you also get High Availability baked in with automatic failover. Introduction to Redis Cluster Tyk can easily speak with a Redis Cluster, or single Redis instance.

For the purposes of this article, and to illustrate my point, we are simply deploying a single default configuration Redis master node on the same tiny Digital Ocean droplet as the gateway.

Tyk Open Source API Gateway

Tyk Gateway requires CPU, a small amount of RAM and fast networking. The Gateway is multi-threaded, and as such, will by default, under load, consume all the CPU made available to it by the host operating system.

We would always recommend compute optimised nodes if deploying to a cloud provider like DO, Linode, AWS, Azure or GCP.  Cloud providers usually provide a good balance of RAM to CPU, and RAM should not really be a concern for Tyk to perform optimally, unless you intend to also run RAM heavy applications on the same node.

Once Tyk Gateways start consuming 60-70% CPU, we would recommend a scaling event. In the event that a node fails or is taken out for maintenance, then the remaining nodes in the cluster should have enough CPU remaining to handle the traffic from the failed node without suffering significant performance degradation.

Installation

We will deploy a commodity DigitalOcean server with 4GB RAM and 2 vCPUs for this benchmark. I will use a one-click-app with Docker deployed on Ubuntu 18.04 for speed of deployment.

Going against everything I have said above, I will be deploying upstream, load server, Gateway and Redis all on the same box.

This post is about how we can easily tune Tyk Gateway performance so the fact that all services are deployed on the same machine (resource contention) only proves that Tyk can go significantly faster if configured appropriately. Moving the target upstream and load generating servers to their own boxes, or ensuring appropriate resource limits if deploying inside Kubernetes for example.

Infrastructure / Application Setup

First, we install & startup NGinX which will be our baseline upstream server – you could use any performant upstream server you like really.

# install nginx
apt install nginx
systemctl start nginx
# Install Redis
apt install redis-server
systemctl start redis-server

# Install Tyk Gateway
apt install curl gnupg apt-transport-https
curl -L https://packagecloud.io/tyk/tyk-gateway/gpgkey | sudo apt-key add -

cat > /etc/apt/sources.list.d/tyk_tyk-gateway.list <<- SOURCES
deb https://packagecloud.io/tyk/tyk-gateway/ubuntu/ trusty main
deb-src https://packagecloud.io/tyk/tyk-gateway/ubuntu/ trusty main SOURCES
apt update
apt install tyk-gateway
systemctl start tyk-gateway

# check its up
curl -i localhost:8080/hello

HTTP/1.1 200 OK
Date: Sun, 03 Feb 2019 22:25:21 GMT
Content-Length: 10
Content-Type: text/plain; charset=utf-8

Hello Tiki

Gateway Proxy Configuration

# set env var to get ip address of eth1 easily
export privateip=$(ifconfig eth1 | grep "inet " | awk '{print $2}')

# configure a keyless API to override sample application
cat > /opt/tyk-gateway/apps/app_sample.json <<- EOF
{
    "name": "nginx",
    "api_id": "nginx",
    "org_id": "default",
    "definition": {
        "location": "",
        "key": ""
    },
    "use_keyless": true,
    "auth": {
        "auth_header_name": ""
    },
    "version_data": {
        "not_versioned": true,
        "versions": {
            "Default": {
                "name": "Default",
                "expires": "3000-01-02 15:04",
                "use_extended_paths": true,
                "extended_paths": {
                    "ignored": [],
                    "white_list": [],
                    "black_list": []
                }
            }
        }
    },
    "proxy": {
        "listen_path": "/nginx/",
        "target_url": "http://$privateip/",
        "strip_listen_path": true
    },
    "do_not_track": true
}
EOF

# reload Gateway configurations via it’s control API
curl -H "x-tyk-authorization: 352d20ee67be67f6340b4c0605b044b7" \ localhost:8080/tyk/reload/

# or just restart the service given that we are not in production
systemctl restart tyk-gateway

# test it
curl -i localhost:8080/nginx/
HTTP/1.1 200 OK
Content-Type: text/html
Date: Sun, 03 Feb 2019 22:38:53 GMT
Etag: W/"5c5747c8-264"
Last-Modified: Sun, 03 Feb 2019 19:58:00 GMT
Server: nginx/1.14.0 (Ubuntu)
X-Ratelimit-Limit: 0
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 0
Content-Length: 612

<!DOCTYPE html>
<html>

--- SNIP ---

As you can see, this is an out-of-the-box Tyk installation.

Setting up benchmarking

In order to benchmark Tyk, we will use docker-hey https://hub.docker.com/r/rcmorano/docker-hey

# fix firewall rules to allow docker0 access to host machine
iptables -A INPUT -i docker0 -j ACCEPT
docker pull rcmorano/docker-hey

Baseline NginX Upstream

# benchmark nginx directly to act as a baseline

docker run --rm -it rcmorano/docker-hey -z 30s http://$privateip/

Summary:
  Total: 30.0029 secs
  Slowest: 0.0305 secs
  Fastest: 0.0001 secs
  Average: 0.0029 secs
  Requests/sec: 17244.7455

Response time histogram:
  0.000 [1] |
  0.003 [303947] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.006 [193680] |■■■■■■■■■■■■■■■■■■■■■■■■■
  0.009 [16511] |■■
  0.012 [2364] |
  0.015 [562] |
  0.018 [202] |
  0.021 [38] |
  0.024 [66] |
  0.027 [20] |
  0.031 [1] |

Latency distribution:
  10% in 0.0007 secs
  25% in 0.0016 secs
  50% in 0.0027 secs
  75% in 0.0040 secs
  90% in 0.0050 secs
  95% in 0.0058 secs
  99% in 0.0083 secs

Details (average, fastest, slowest):
  DNS+dialup: 0.0000 secs, 0.0001 secs, 0.0305 secs
  DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
  req write: 0.0000 secs, 0.0000 secs, 0.0225 secs
  resp wait: 0.0026 secs, 0.0000 secs, 0.0241 secs
  resp read: 0.0002 secs, 0.0000 secs, 0.0220 secs

Status code distribution:
  [200] 517392 responses

So, out of the box, NginX performs as follows – even though docker-hey is also consuming CPU and eating system resources.

Slowest 30.5ms
Average 2.9ms
Median 2.7ms
95%ile 5.8ms

Baseline Tyk Reverse Proxy to NginX

So now that we have our baseline, we should see how Tyk out of the box performs as a transparent reverse proxy to NginX – even with some resource contention.

docker run --rm -it rcmorano/docker-hey -z 30s http://$privateip:8080/nginx/

Summary:
  Total: 30.0035 secs
  Slowest: 0.0768 secs
  Fastest: 0.0003 secs
  Average: 0.0099 secs
  Requests/sec: 5056.8813

Response time histogram:
  0.000 [1] |
  0.008 [67948] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.016 [58333] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.023 [22119] |■■■■■■■■■■■■■
  0.031 [2519] |■
  0.039 [544] |
  0.046 [198] |
  0.054 [33] |
  0.062 [18] |
  0.069 [6] |
  0.077 [5] |

Latency distribution:
  10% in 0.0034 secs
  25% in 0.0055 secs
  50% in 0.0087 secs
  75% in 0.0135 secs
  90% in 0.0177 secs
  95% in 0.0203 secs
  99% in 0.0268 secs

Details (average, fastest, slowest):
  DNS+dialup: 0.0000 secs, 0.0003 secs, 0.0768 secs
  DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
  req write: 0.0000 secs, 0.0000 secs, 0.0022 secs
  resp wait: 0.0098 secs, 0.0002 secs, 0.0768 secs
  resp read: 0.0000 secs, 0.0000 secs, 0.0054 secs

Status code distribution:
  [200] 151724 responses
Baseline Latency Latency (ms) Introduced Latency (ms)
Slowest 30.5 76.8 +46.3
Average 2.9 9.9 +7
Median 2.7 8.7 +6
95%ile 5.8 20.3 +14.5

Not bad – Tyk is able to handle 5000 rps out of the box despite the resource contention – this tiny machine is working very hard and all the CPU is being consumed.

The problem we have here – is that our test is extremely aggressive – the average/median latency is acceptable, however there are some outliers which causes average latency to increase. The slowest request introduces 46ms latency which for me is unacceptable.

Resource Contention

Now in a real world scenario, you simply wouldn’t hammer the Gateway – the added latency for some of the requests is just not acceptable. Let’s throttle our rps a little.

By default, docker-hey has 50 concurrent clients. The Gateway can handle 5k rps max in this setup. So by turning down the requests per second by just 5%, let’s see how latency improves, ensuring that requests come back in a timely and consistent fashion. We will set the rate limit per consumer to 95.

Rate limit per consumer = 5000/50 * 95% = 95
docker run --rm -it rcmorano/docker-hey -z 30s -q 95 http://$privateip:8080/nginx/

Summary:
  Total: 30.0138 secs
  Slowest: 0.0506 secs
  Fastest: 0.0002 secs
  Average: 0.0069 secs
  Requests/sec: 4713.9667

Response time histogram:
  0.000 [1] |
  0.005 [58984] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.010 [57735] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.015 [18420] |■■■■■■■■■■■■
  0.020 [5468] |■■■■
  0.025 [740] |■
  0.030 [110] |
  0.036 [22] |
  0.041 [2] |
  0.046 [1] |
  0.051 [1] |

Latency distribution:
  10% in 0.0027 secs
  25% in 0.0040 secs
  50% in 0.0060 secs
  75% in 0.0090 secs
  90% in 0.0127 secs
  95% in 0.0150 secs
  99% in 0.0192 secs

Details (average, fastest, slowest):
  DNS+dialup: 0.0000 secs, 0.0002 secs, 0.0506 secs
  DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
  req write: 0.0000 secs, 0.0000 secs, 0.0032 secs
  resp wait: 0.0069 secs, 0.0002 secs, 0.0506 secs
  resp read: 0.0000 secs, 0.0000 secs, 0.0121 secs

Status code distribution:
  [200] 141484 responses

Whilst we lost just shy of 350 requests per second, look what we gained!

Baseline Latency Latency (ms) Introduced Latency (ms)
Slowest 30.5 50.6 +20.1
Average 2.9 6.9 +4
Median 2.7 6.0 +3.3
95%ile 5.8 15.0 +9.2

Latency is now at a much more acceptable level – by not overloading the CPU. Notice how we have still not actually done any tuning, but have still managed to reduce CPU consumption by more than 10%.

Tyk Gateway is written in Go

Unlike various API Management solutions Tyk is written in pure Go. Go is a modern systems programming language, and is being actively developed by Google, to solve Google scale problems. Tyk does not rely on 3rd party software such as NginX to do all the heavy lifting. Go provides both Type safety, and also offers a Garbage Collector.

The Go programming language has an enviably performant concurrent, tri-colour, mark sweep garbage collector first proposed by Edsger W. Dijkstra in 1978. If you want the details, https://dl.acm.org/citation.cfm?id=359655. At a very high-level, stop-the-world GC pauses are now well below 10ms.

Despite the awesome GC, at such high throughput, compute is now being spent on scheduling the garbage collector, and collecting that garbage.

At the expense of a little extra RAM usage, we can tweak the scheduler so that it runs less often. The default value is 100%, and each time we double it, the collection frequency is halved. I’m going to set the value to 3200%. This can be achieved simply by setting a system environment variable GOGC.

It’s worth playing around with this figure for specific use-case – don’t set it too high, or you will run out of RAM and crash the VM. There will come a point where you don’t get noticeable gains. I doubled the GOGC value each test to find what the optimum figure was and that’s how I arrived at the 3200% figure.

Tuning Tyk’s Garbage Collector

vi /lib/systemd/system/tyk-gateway.service

# add this line inside the [Service] directive
Environment="GOGC=3200"


# save, quit and restart daemon and service
systemctl daemon-reload
systemctl restart tyk-gateway

Let’s re-run the benchmark

docker run --rm -it rcmorano/docker-hey -z 30s -q 95 http://$privateip:8080/nginx/

Summary:
  Total: 30.0065 secs
  Slowest: 0.0425 secs
  Fastest: 0.0003 secs
  Average: 0.0048 secs
  Requests/sec: 4739.7088

Response time histogram:
  0.000 [1] |
  0.004 [69149] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.009 [69266] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.013 [2489] |■
  0.017 [841] |
  0.021 [373] |
  0.026 [67] |
  0.030 [16] |
  0.034 [4] |
  0.038 [9] |
  0.043 [7] |

Latency distribution:
  10% in 0.0027 secs
  25% in 0.0035 secs
  50% in 0.0045 secs
  75% in 0.0056 secs
  90% in 0.0067 secs
  95% in 0.0077 secs
  99% in 0.0126 secs

Details (average, fastest, slowest):
  DNS+dialup: 0.0000 secs, 0.0003 secs, 0.0425 secs
  DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
  req write: 0.0000 secs, 0.0000 secs, 0.0045 secs
  resp wait: 0.0047 secs, 0.0002 secs, 0.0425 secs
  resp read: 0.0000 secs, 0.0000 secs, 0.0052 secs

Status code distribution:
  [200] 142222 responses

Holy Cow – we just dropped from 82% CPU usage down to less than 60%!

Holy Cow – we just dropped from 82% CPU usage down to less than 60%! We are now using 26% less CPU than the previous test and about 40% less CPU than our first bench.

Tyk is still handling the same RPS, but rather than waste compute on collecting garbage, the Gateway is being able to proxy requests.

Baseline Latency Latency (ms) Introduced Latency (ms)
Slowest 30.5 42.5 +12
Average 2.9 4.8 +1.9
Median 2.7 4.5 +1.8
95%ile 5.8 7.7 +1.9

The slowest request is just 12ms slower than the slowest baseline request. Tyk is now adding sub 2ms 95%ile latency.

At 60-70% CPU usage we would recommend adding another Tyk to the cluster. But this is a benchmarking blog – so let’s up the RPS. Let’s try full speed again to see what Tyk can handle when maxing out.

docker run --rm -it rcmorano/docker-hey -z 30s http://$privateip:8080/nginx/

Summary:
  Total: 30.0050 secs
  Slowest: 0.0955 secs
  Fastest: 0.0003 secs
  Average: 0.0072 secs
  Requests/sec: 6897.5780

Response time histogram:
  0.000 [1] |
  0.010 [165543] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.019 [38325] |■■■■■■■■■
  0.029 [2665] |■
  0.038 [332] |
  0.048 [75] |
  0.057 [6] |
  0.067 [8] |
  0.076 [2] |
  0.086 [3] |
  0.096 [2] |

Latency distribution:
  10% in 0.0030 secs
  25% in 0.0046 secs
  50% in 0.0066 secs
  75% in 0.0091 secs
  90% in 0.0120 secs
  95% in 0.0144 secs
  99% in 0.0212 secs

Details (average, fastest, slowest):
  DNS+dialup: 0.0000 secs, 0.0003 secs, 0.0955 secs
  DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
  req write: 0.0000 secs, 0.0000 secs, 0.0042 secs
  resp wait: 0.0072 secs, 0.0002 secs, 0.0955 secs
  resp read: 0.0000 secs, 0.0000 secs, 0.0059 secs

Status code distribution:
  [200] 206962 responses
  req write: 0.0000 secs, 0.0000 secs, 0.0078 secs
  resp wait: 0.0072 secs, 0.0002 secs, 0.0684 secs
  resp read: 0.0000 secs, 0.0000 secs, 0.0069 secs

Status code distribution:
  [200] 205300 responses

So the Gateway is now handling 6900 rps rather than the original 5000 rps (27% increase), and latency is still pretty good adding just 8.6ms at the 95th %ile – That’s a massive 40% improvement in latency from the out-of-the-box installation of Tyk.

Baseline Latency Tyk Latency (ms) Introduced Latency (ms)
Slowest 30.5 95.5 +65.5
Average 2.9 7.2 +4.3
Median 2.7 6.6 +3.9
95%ile 5.8 14.4 +8.6

By dialling back the throughput once again by about 5% so that we are not overloading the CPUs – we set q to 131 (6900/50*0.95).

docker run --rm -it rcmorano/docker-hey -z 30s -q 131 http://$privateip:8080/nginx/

Summary:
  Total: 30.0058 secs
  Slowest: 0.0473 secs
  Fastest: 0.0003 secs
  Average: 0.0052 secs
  Requests/sec: 6412.0916

Response time histogram:
  0.000 [1] |
  0.005 [105806] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.010 [76053] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.014 [7891] |■■■
  0.019 [1834] |■
  0.024 [563] |
  0.028 [194] |
  0.033 [52] |
  0.038 [4] |
  0.043 [1] |
  0.047 [1] |

Latency distribution:
  10% in 0.0027 secs
  25% in 0.0036 secs
  50% in 0.0047 secs
  75% in 0.0060 secs
  90% in 0.0079 secs
  95% in 0.0100 secs
  99% in 0.0158 secs

Details (average, fastest, slowest):
  DNS+dialup: 0.0000 secs, 0.0003 secs, 0.0473 secs
  DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
  req write: 0.0000 secs, 0.0000 secs, 0.0026 secs
  resp wait: 0.0051 secs, 0.0002 secs, 0.0473 secs
  resp read: 0.0000 secs, 0.0000 secs, 0.0034 secs

Status code distribution:
  [200] 192400 responses
Baseline Latency Total Latency (ms) Tyk Introduced Latency (ms)
Slowest 30.5 47.3 +17
Average 2.9 5.2 +2.3
Median 2.7 4.7 +2.0
95%ile 5.8 10.0 +4.2

CPU is at a much healthier 80%, we are still handling 6400 requests per second (1400 rps more than original) and are introducing just 2.3ms average latency or 4.2ms latency at the 95th %ile. Therefore a 54% improvement on the default Tyk Installation.

Summary

It is important to remember that everything has been deployed on a 2-virtual-core commodity server with just 4GB of RAM.

Because we still have significant resource contention from the upstream server, load server, Redis and the Gateway all deployed on the same box, there is no way of knowing the true performance of the gateway without splitting out the components onto their own machines, or dedicating compute to each component.

Despite all of this, by tweaking a single environment variable, Tyk has been able to handle 6400 requests per second. 95% of requests come back within 10ms and subtracting NginX’s baseline latency, Tyk has introduced just 4.2ms 95th %ile latency to the request.

To top that, all the services combined are only consuming 80% CPU and less than 600Mb RAM. Further optimisations can be made to Tyk configurations to tune Tyk further in accordance with your specific use-case.

Want to give it a go? We’d love to hear your results! Drop us a line via the Tyk Community to share your own benchmarks, and help us continue to improve Tyk performance.

Happy benching!

© Tyk Technologies, 2019