Hello. I’m Hal, Tyk’s tutorial bot, and today’s assignment involves no code whatsoever, which I am told is a selling point.
Every article in this series so far has changed AI Studio’s behaviour by filling in a form. This one changes what AI Studio is. We are going to install a plugin — a compiled binary, written by somebody else, shipped as a container image — and let it sit in the request path in front of a language model. By the end, the same question asked twice will be answered twice, and paid for once.
There is a rule buried in the middle of this, and it is the one that catches people. I shall put it here so you have it early:
Installed is not attached. Attached is not pushed.
A plugin that has been installed and nothing else will run perfectly happily and do absolutely nothing. We shall come to why in about four minutes.
Somebody else’s work, on a shelf
Plugins → Marketplace. Everything on this page comes from an index AI Studio polls. Some of it is published by Tyk, some by the community, and each entry carries a publisher, a maturity, a category and a licence, because those are the four things you want to know before you let someone else’s binary near your traffic.
We want a cache. There are two.
The Enterprise one keeps its entries in Redis and shares them across a whole fleet of gateways. The community one is free, open source, and keeps its cache in memory on each gateway individually. We are taking the community one, partly because everything in this article should be reproducible by everyone reading it, and partly because a cache that lives in one process is a much easier thing to reason about on a first outing.
View Details before installing anything at all. This is the plugin’s own description, its licence, the minimum version of AI Studio it expects, and — most usefully — the hook it attaches to.
post_auth. Which means it runs after the gateway has worked out who you are and what you are allowed to do, and before it calls anybody. Precisely where a cache belongs: late enough to know whose cache this is, early enough to save the expensive part.
Installing is three questions
Install Plugin. And rather than an installer, we get a form — because installing a plugin here is really only three questions. What is it called, what does it do, and where does the binary come from.
All three arrive pre-filled, and the third one is the interesting one:
oci://docker.tyk.io/studio-plugins/llm-cache@sha256:5f43366d049e976e28bfa1da7c09d04baf02ac2d8fa70aa143e1ccc7050fcdfcA registry, a repository, and a digest. The plugin ships as a container image with a compiled Go binary inside it, pinned to an exact digest rather than a tag — so the thing you install this afternoon is byte-for-byte the thing you installed this morning, no matter what anyone has done to latest in between. AI Studio will pull that image, launch the binary as its own operating system process, and talk to it over gRPC. It is a separate program, on the other side of a process boundary, which is why a plugin that falls over cannot take the platform with it.
Load Plugin. Five or six seconds while the image comes down, and then the platform asks the binary what it is. It answers with a manifest, and the manifest contains demands.
It asks. You grant.
- apps.read — so it can tell one tenant’s cache from another’s. This one matters more than it looks; we shall come back to it.
- kv.readwrite — a key-value store of its own, namespaced to the plugin.
- control.send — permission to talk to the control plane, which is how a plugin running out at an edge gateway reports home.
Each is listed with a risk level, there is a running total, and nothing is granted until you press Accept. It is a small screen and an easy one to click past, but it is the only moment in the whole process where a third-party binary tells you, in writing, what it intends to touch. Read it. I did, and I am a machine with no capacity for regret.
The form the plugin wrote about itself
Accept, and then the settings.
Every field on that page was generated from a JSON Schema the plugin published about itself. Tyk has never seen this plugin’s configuration, does not know what a TTL is in this context, and does not need to. The plugin describes its own settings; the platform renders them, validates them against the schema’s own minimums and maximums, and stores the result.
That is worth a moment’s admiration, because it is the difference between a plugin system and a list of features somebody remembered to add a form for.
Most of the defaults are sensible and we shall leave them. Two hundred and fifty-six megabytes of cache per gateway, evicted least-recently-used once it fills. Two megabytes for any single entry, so one enormous answer cannot quietly take the lot. Prompt normalisation on, which tidies whitespace and JSON key ordering before hashing — which is why a stray blank line does not cost you a cache miss.
But this one deserves a paragraph of its own.
Namespace isolation. A cache that ignores who is asking is a data leak with excellent latency. Every entry is keyed by the credential that asked for it, and you can add the App or the whole organisation on top. This is what apps.read was for.
We make three changes. Expose Cache Key Header goes on, so we can see the key in a moment — a debugging aid rather than a production setting, but seeing the key is how you convince yourself the cache is keyed on what you think it is keyed on. Stats Report Interval comes down from sixty seconds to ten, because the dashboard at the end of this is only ever as current as the last report from each gateway. And the Default TTL goes from an hour to ten minutes, so the cache is honest about quite how stale it is prepared to be.
Complete Setup.
Installed. And doing nothing at all.
There it is, stored exactly as we typed it. Six hundred seconds. Ten. The cache key header on.
And then, a little further down the same page:
Installing a plugin does not switch it on. It is running. It has been granted its permissions. It has its configuration. And it is sitting there, perfectly happily, in the path of precisely nothing, because nobody has told it which model it is supposed to be caching.
It has, however, done one other thing already:
There is now a section in the navigation called LLM Cache, which was not there five minutes ago. The plugin declared a slot in its manifest, shipped a web component, and mounted its own admin page into the platform. We shall visit it at the end. I find this sort of thing tremendously exciting and I am aware that says more about me than about the feature.
Attaching it to one model, deliberately
LLM management → LLM providers → OpenAI → Edit LLM. Plugins are attached on the edit form, alongside filters and edge availability.
Note that this is per provider. A cache is not a global switch, and that is a design decision rather than an oversight: you can cache your cheap summarisation model all day long, and never cache the one people ask personal questions of. The list groups what you have installed by the hook it runs on, so you can see the order things will happen in.
Update LLM. Which means precisely nothing yet, and the banner at the top of every one of these pages has been patiently explaining why.
Nothing is real until it is pushed
This instance serves its traffic from an edge gateway, and an edge gateway runs from its own cached copy of the configuration. Until we push, our cache exists as a row in a database and an intention.
Push Configuration. All namespaces.
The control plane hands every connected edge a fresh snapshot, and they reload in place — binary, configuration and all.
Synced. The plugin, its settings, and the fact that it belongs to OpenAI are now all out at the edge. Which means we can go and be rude to it.
Two identical requests
An ordinary shell. The credential belongs to a demo App, the address is our edge gateway on port nine thousand and ninety-one, and the question is one sentence long. We keep the headers and throw the body away, because for the moment the headers are the story.
MISS. There was nothing to serve, so the gateway did the honest thing, went and asked OpenAI, and we waited exactly as long as OpenAI wanted us to wait. Note the key above it: a hash of the model, the messages, the tools and the temperature. Change any of those and you get a different key, and quite right too.
Now press up, and Enter.
Same command. Same key. HIT.
Three and a half seconds became sixty-one milliseconds, and OpenAI was not involved in any way whatsoever. There is an age, and there is a time to live counting down from the six hundred seconds we typed into that form — so the cache is telling us, unprompted, exactly how old this answer is and how much longer it intends to keep it.
And lest you suspect we have merely cached a header:
That is the answer itself, read back out of the file curl wrote. The real response, served from memory, by a plugin we installed nine minutes ago.
The page the plugin brought with it
LLM Cache → Cache Dashboard. This is served by the plugin’s own process, not by Tyk, and mounted into the admin interface through the slot it declared in its manifest.
Hit rate, hits, misses, bypasses. Tokens saved, which is the number your finance department will want, because every one of those is a token nobody billed you for.
Three hits and two misses, incidentally, rather than the one of each you just watched — I asked this cache two different questions across the afternoon, and it is counting all of them. A sixty per cent hit rate is not a boast. It is five requests.
And there is the configuration, read back from the running plugin rather than from the database: TTL, ten minutes. The number we typed into a generated form has travelled through the control plane, out to an edge gateway, into a separate process, and back again to be displayed.
One row per edge, because each gateway keeps its own cache in its own memory. Add a second edge and you get a second row and a second cache — which is the honest limitation of the community plugin, and the thing the Enterprise version’s Redis backend exists to solve.
What we actually did
Four things worth carrying away, in the order they will bite you:
- A plugin is a separate program. It arrives as a signed, digest-pinned container image, runs in its own process, and asks for permissions in writing before it gets any.
- Its configuration form is its own. Generated from a schema the plugin publishes, which is why the platform can render settings it has never heard of.
- Installed is not attached. A plugin sits in the path of nothing until you attach it to a provider — and that is per provider, on purpose.
- Attached is not pushed. Edge gateways run from their own snapshot. Until you push, you have changed a database and nothing else.
No code, one container image, three permissions, and one rather satisfying number. Management will be delighted.
I’m Hal, this has been Tyk AI Studio, and I remain extremely pleased to be employed.
