The drivers behind the transformation
Sarah, a lead platform engineer at one of America’s largest financial institutions, faced a critical challenge. Thousands of field reps relied on internal APIs to serve customers, but reactive monitoring wasn’t cutting it. Too much time was lost by the platform team fighting fires when it could have been spent building capabilities.
The opportunity was clear and so were the business drivers. The company needed to:
- Maximize uptime for field reps relying on internal APIs
- Detect breakages early and trigger a fast team response
- Establish clear golden signals for healthy vs. unhealthy states
- Reduce incident chaos by ensuring actionable alerts
- Continuously improve reliability through post-incident learnings
Key to achieving this was a change of mindset: The organization now treats alerting as a strategic capability that directly supports business operations. Clean dashboards and manageable incident reports benefit not just Sarah and her team but thousands of field reps and the customers they support.
Here’s how they did it…
Alerting at enterprise scale
When you’re processing tens of millions of API calls daily – from mobile banking to compliance reporting – traditional alerting breaks down fast. At this scale, every alert needs to be actionable, every incident needs context, and every resolution must happen before customers notice.
The golden signal revolution
The breakthrough came when Sarah’s platform engineering team focused on “golden signals” – metrics that directly correlate with user experience and business impact.
Traffic patterns that make sense. The revolution began with the system identifying a range of normal traffic fluctuations: Monday mornings see 300% higher API traffic; market openings trigger predictable spikes; cafeteria ordering peaks before lunch. With usual traffic patterns identified, the system could stop alerting on normal business fluctuations, reducing unnecessary noise.
Latency where it matters. The team then monitored 95th and 99th percentile latency across critical endpoints. If the 99th percentile creeps toward two seconds, the team knows trouble is brewing. This has enabled them to catch performance issues 20-30 minutes before customer impact.
Errors in context. A 5% error rate during a DDoS attack might be heroic. A 0.5% bump in authentication failures during normal hours signals a security incident. Context became everything as the team evolved the alerting system into something more strategic.
Building alerting as a product
A key element of the mindset shift was that the team began to treat alerting as a product requiring continuous improvement, not a one-time project. As part of this, post-incident learning loops became standard. After every incident, the team asked: What alert would have helped? Which alerts weren’t actionable? How do we respond faster?
They also replaced static thresholds with dynamic tolerance bands. This accounted for instances such as fraud detection APIs handling higher error rates during attacks, and payment systems maintaining tighter tolerances during peak hours. Quarterly calibration ensures the thresholds evolve with the business.
The technology foundation
Tyk API Gateway serves as the organization’s central nervous system – the single source of truth for API health. Built-in analytics track golden signals across all traffic without overhead. Key to this are:
- Synthetic monitoring: Continuous testing of critical user journeys every one to five minutes. This catches issues before real users experience them.
- Intelligent alert routing: Minor issues at 2 pm go to Slack. The same issue at 2 am pages the on-call engineer directly.
The results: What success looks like
Organizations that successfully transform their alerting strategies typically see a range of quantifiable improvements – and this global financial services company was no exception:
- Mean time to detection (MTTD): Often reduced from 20+ minutes to under five minutes
- Mean time to recovery (MTTR): Frequently cut by 50-70%
- Alert volume: Decreased by 60-80% while catching more actual issues
- Customer-impacting incidents: Reduced by 40-60% year-over-year
Strategic alerting also delivers a significant human impact. Engineering teams benefit from faster detection and more targeted responses, while clearer, actionable signals reduce the chaos that once defined incident management. The continuous refinement process means reliability keeps improving over time, creating a positive cycle in which each incident makes the system stronger.
Where AI fits
Sarah’s team took a pragmatic approach to integrating alerting with the organization’s AI initiatives. Their focus was on:
- Anomaly detection: Machine learning establishes dynamic baselines, but humans make final decisions on thresholds and escalation.
- Pattern recognition: AI correlates signals across infrastructure layers. When payment APIs show elevated latency, the system automatically checks related database performance and network metrics.
- Predictive alerting: They’re experimenting with models that forecast issues based on historical patterns, providing earlier warnings rather than replacing human judgment.
The regulatory reality
Financial services require documentation of every system change, as well as incident reporting to regulators and quantification of customer impact. This organization’s alerting strategy now supports both operational excellence and regulatory compliance through detailed audit trails and automated incident documentation.
“‘Good enough’ isn’t good enough,” their CTO explains. “We need provably reliable systems. That means open standards, established practices, and partners with enterprise expertise.”
Lessons learned
The financial services organization emphasized four key lessons learned from the process of turning alerting chaos into calm:
Start with golden signals. “We tried monitoring everything initially,” explains their VP of Platform Engineering. “The breakthrough came with focusing on metrics that predicted customer impact.”
Plan for blind spots. The business implemented “silence alerts” – notifications that occur when expected logs or metrics stop appearing. This helps prevent them from being caught unaware.
Culture complements technology. The organization didn’t rely on tech alone. It also invested heavily in training, documentation, and runbooks to ensure alerts lead to effective action.
Continuous testing is essential. After implementation, the team would revisit incidents regularly and refine those signals to detect issues earlier and engage the right teams faster.
The partnership advantage
Tyk’s commitment to open standards and seamless integration enabled rapid transformation. “We didn’t want to rip and replace our monitoring stack,” Sarah explains. “Tyk integrated with our existing systems, so we could swiftly leverage additional value from them.”
Crucial to maximizing this success was a partnership approach. Increasingly, trusted partner relationships are delivering notable advantages compared to traditional client-vendor interactions. It’s not only something this organization has learned but a message that the Tyk team is hearing from financial services leaders around the world. To discover what else is top of mind for financial leaders right now, head over to this article.
You can also chat with the Tyk team about how a partnership can underpin your strategic success, whether in relation to alerting or anything else API-related.
