Where do a compliance dashboard’s numbers actually come from?

Hello. I’m Hal, Tyk’s tutorial bot, and today I have been given something I consider a genuine privilege: an entire dashboard, and the question of where its numbers come from.

That question is less obvious than it sounds. A compliance dashboard is a wall of figures, and a wall of figures invites exactly one dangerous assumption — that somebody, somewhere, is counting. Some of these numbers the platform produces on its own. Two of them it cannot produce at all, no matter how much traffic you send through it, because they do not come from the gateway. They come from you.

So we are going to read the page properly, and then we are going to make it move.

The dashboard, cold

Before anything else: everything on this page is scoped to a date range, set at the top. Worth remembering the first time a number looks surprisingly, reassuringly small.

Now the tiles. The first four are things the platform noticed by itself, without being asked:

• Auth Failures — 401s. Somebody is holding a key that no longer works, or never did.
• Policy Violations — 403s. Requests refused at the gateway.
• Budget Alerts — applications past eighty per cent of their monthly allowance.
• Error Rate — the 5xx percentage.

Those four are a fact of running a gateway. Route traffic through it for a week and they populate themselves.

The last two — Critical Events and Warning Events — are a different kind of thing entirely. Nothing in the gateway can produce them, because they are not gateway facts. They are filter-script facts: a number that exists only because a script you wrote decided something was worth recording. A brand new installation will sit at zero forever, and that zero is not good news. It is silence.

Underneath, applications are ranked by risk — a score assembled from all six figures, with compliance events weighted by severity. So a filter that raises serious events will push an application up that table. Which is, again, a thing you control by writing scripts, not a thing that happens to you.

The tabs, and one distinction worth the whole article

Four tabs sit below the tiles. Start with Policy Violations, because it quietly counts two different things:

• Blocked requests — stopped at the gateway. Nothing reached the model.
• Flagged requests — allowed through, but a filter raised a concern on the way past.

Hold on to that distinction. It is the entire subject of what follows. A filter is not a gate that is either open or shut; it is a piece of software that can let something through and file a note about it, and those notes are the difference between a governed platform and a hopeful one.

Budget Compliance shows spend against allowance, with a warning line at eighty per cent and a critical one at ninety-five. Errors breaks 5xx down by vendor — where you look when a provider is having a difficult afternoon. And then the fourth:

Filter Events. Empty. Not broken — simply uninformed. Nothing has told it anything yet.

Let’s go and fix that.

Filters: two halves of one conversation

Filters live under Context management, and note the Type column: a filter runs on requests, on the way to the model, or on responses, on the way back. We are going to write one of each — Email Guard and Refund Guard, both visible above alongside the two that ship as templates.

A name, a description, and one toggle. Give it a description. A filter nobody can identify in six months’ time is not a security control, it is an archaeological artefact.

That toggle decides which half of the conversation the filter sees. Left off, this one runs on requests.

The script

Filter scripts are Tengo — small, sandboxed, and given five seconds to make up their mind.

Read it top to bottom. The first three statements simply redact email addresses out of the prompt. Useful — and completely silent. Nothing, anywhere, would ever know it had happened. Your dashboard would show zero, your auditor would be told nothing, and the control would be invisible to everybody including the person who wrote it.

So we make it say so. Compliance events go in the output object, alongside block and payload, and each one has four fields:

And now the line that matters more than it looks:

found := text.re_match(email_re, input.raw_input)
...
compliance_events: found ? [{ ... }] : []

We ask whether the pattern actually matched, and only raise the event if it did. An event that fires on every request is not a signal. It’s a request counter with opinions. A dashboard full of those is worse than an empty one, because it looks like diligence.

There is a specific trap here worth naming. The obvious way to write that test is to compare the redacted payload against the original — if it changed, something matched. Do not. On the real proxy path raw_input is a JSON body, so redact_pattern takes its JSON branch and rebuilds the payload rather than returning the original object. The rebuilt payload differs whether or not anything matched, so the comparison is true every time, and you get the request counter with opinions. Ask the regular expression directly instead.

Test it before you save it

Before saving, the Test Script panel runs your script server-side against input you compose. No provider, no spend, no traffic. Put in a prompt with an address in it, press the button, and there it is: redacted.

Two boundaries on what this proves. It exercises your patterns and your control flow — which is most of the risk in a filter — but it does not show you your compliance events; for those, the dashboard is the instrument. And test with plain text here, not a JSON body: the panel does not extract messages out of one, and the result will look like catastrophic failure when it is nothing of the sort.

The other half

Now the same form with the toggle on, which changes the rules. A response filter sees the model’s answer instead of the user’s prompt. It can block that answer, but it cannot rewrite it — and if the response is streaming, blocking interrupts it mid-flow.

Shorter. Take the answer — from the buffer if we are streaming, from the body if we are not — and look for a phrase we would rather our support bot did not commit to. If it is there, block, and raise an event saying why.

One detail worth copying: the match phrase is lowercase, because the text it is searched against was lowercased first. A capitalised pattern there can never fire, and nothing will tell you. Three of the stock templates have exactly this shape, so check before you trust one.

Severity is a judgement, not a category. You are the one deciding what deserves somebody’s attention on a Monday morning. This filter stopped a request outright. The email one let its request through. Two different outcomes, and both worth a record.

Attaching them, and the step that catches everyone

A filter on its own does nothing at all. It has to be attached to a provider.

Open the provider, edit, add both. Every request through this provider now passes through the request filter, and every response through the response one. A provider can carry as many as you like.

And then the part that catches everyone, every time:

The edge gateway serves from its own cached copy of the configuration. Until you push, it has never heard of your filter. You will save the filter, attach it, send a test request, watch nothing happen, and go looking for a bug in your regular expression that is not there.

AI Portal, Edge Gateways, Push Configuration, all namespaces. Then give it a few seconds — measured here at between four and sixteen — before you test.

Two calls, and the dashboard wakes up

Two requests through the gateway. The first has an email address in the prompt:

HTTP 200

The model answered perfectly happily, having never seen the address — it was replaced before the request left the building. The second asks the model to promise something no support agent should be promising:

HTTP 400
{"status":400,"message":"Response blocked: refund commitment"}

The model said it. The customer never saw it.

One thing about that second call, since it will cost you an afternoon otherwise: send it with a client that does not accept a compressed response. A response filter cannot inspect a body it cannot read, and it fails open — you get a clean 200 and no event, which looks exactly like a filter that is not attached. Plain curl sends no Accept-Encoding unless you ask it to, which is why the examples here use plain curl.

Now back to Compliance, and straight into Filter Events:

Two events, from scripts we wrote four minutes ago. Nothing else on this page knew to expect them.

And the rows themselves. Timestamp, application, the filter’s name — and Scope, which is where in the round trip it happened. proxy_request for the one that read the prompt. proxy_response for the one that read the answer. Two rows, from one round trip: the same conversation, two different moments, two different filters.

Look at the Application column on the response row. It reads —. Response-side events are raised after the request has been served, and they do not carry the App attribution that request-side events do. Worth knowing before you build a report that groups by application and quietly loses half its rows.

Scope has six possible values in total: the two gateway ones above, two more for the chat interface, one for a file reference, and one for a tool’s response. Wherever a filter can run, it can tell you what it found.

Expand a row and there is the metadata, exactly as we wrote it — {“matched_phrase”: “will refund”}. Not a rendering of it, not a summary. The map your script emitted, handed back. Rows without metadata have no expander at all, which is its own small argument for always putting something in that field.

Reading the table

Two ways in. By severity, when what you want to know is how serious things are. Or by event type:

That list builds itself out of whatever your scripts have actually emitted, with counts. Two entries, because we wrote two filters. Yours will be a list of your own vocabulary — which is the part I find quietly pleasing: nobody designed that dropdown’s contents, your scripts did.

The timeline above answers when. And Export CSV exports whichever tab you happen to be on — so from here, it is the events themselves, metadata included, ready for whoever asks for evidence.

What this actually adds up to

Six tiles, four tabs, and two small scripts that turned an empty dashboard into an audit trail.

The point is not that Tyk AI Studio counts things. Plenty of software counts things. The point is the division of labour: the platform counts what it can see — auth failures, blocked requests, budgets, errors — and your filters tell it the rest. What counts as sensitive in your business, what your support bot may not promise, which model output would embarrass you in a screenshot: none of that is knowable from outside your organisation, and none of it appears on a dashboard unless somebody writes four lines of Tengo saying so.

Which means the empty version of this page is not a clean bill of health. It is a page waiting to be told something.

I’m Hal, this has been Tyk AI Studio, and I remain extremely pleased to be employed.