Monitoring AI After Launch: SLA and Alert Template for Hallucinations, Latency, and Cost

Three weeks after launch, the customer's IT director called at midnight. Their RAG-based system had given wrong information about a return policy. The system had stated a seven-day consideration period as thirty days of unconditional returns. A customer took a screenshot and filed a complaint. The model itself wasn't the problem. What failed was observability. No one caught the error until after a customer reported it. When the system went live, everyone celebrated. Monitoring stopped at the basic question: is the service still running. API responses showed 200 status codes, so the team assumed nothing was wrong.

This gap appears in most AI deployments we handle. Traditional system monitoring tracks availability. AI systems require tracking beyond that. Three specific things matter: whether the model generates false information, whether response times are degrading, and whether token costs are accelerating. Traditional APM misses all three.

Monitoring SLA requires rethinking for AI systems

Traditional backend services have simple health indicators: response codes, latency percentiles, error rates. AI systems introduce a different requirement: verifying that outputs are correct. A successful HTTP response with 800ms latency could contain entirely fabricated information. This means the standard operations playbook doesn't apply. Three metric categories require simultaneous tracking: quality (factual accuracy and refusals), performance (latency), and cost (token consumption). These interact poorly. Reducing model size cuts costs but increases hallucinations. Adding retrieval reranking reduces hallucinations but increases latency. Monitoring translates these tradeoffs into measurable numbers, rather than waiting for customer complaints.

Every metric category needs three components: a measurement method, an SLA target, and an alert threshold. Without all three, monitoring is incomplete.

SLA template for three metric categories

The template below represents our standard starting point for typical business applications. The thresholds are conservative for customer service and knowledge-lookup systems. Adjust based on your risk profile. Internal tools without external impact can use looser thresholds. Financial services and healthcare, where incorrect answers create liability, require tighter bounds.

A few implementation points matter because we learned them the hard way.

Manual spot-checks don't scale for hallucination monitoring. Reviewing dozens of responses per day by hand provides no statistical picture once traffic grows. The approach we use is to deploy an independent LLM as a judge. Every response gets compared against the source documents it claims to cite. The question is simple: did this statement actually appear in the retrieved content? Sample 5 to 10 percent of responses in real time, keep the computational cost manageable, and maintain a rolling groundedness metric. Keep the judge model's prompt separate from the main system in version control. If the main pipeline changes and the judge's prompt drifts with it, the scoring criteria shift without notice, and metrics become unreliable.

Measure latency by stage, not just total response time. A retrieval-augmented generation request involves retrieval, reranking, and generation stages. Without visibility into each stage, bottlenecks stay hidden. Separate time to first token (TTFT, the delay before users see output) from end-to-end completion time. TTFT determines perceived responsiveness in chat interfaces. Users accept longer total times if text streams continuously.

Monitor cost anomalies, not just total spending. Routine traffic doesn't drain budgets. The problem is a prompt fed an entire PDF attachment or a retry loop that multiplies token consumption. Thresholds work best as relative measures (daily average cost exceeds baseline by 1.5 times) rather than absolute values. This catches problems earlier than tracking the total bill.

Thresholds require continuous calibration after launch

The template is an initial baseline, not final settings. During the first two weeks after launch, expect noisy alerts. A P95 spike at midnight might be a batch job. A spike in dislike rates might indicate one user filing multiple similar complaints. Rather than disabling alerts, adjust them. Distinguish between real signals and noise. Refine thresholds and grouping logic until each alert indicates something actionable. A two-tier approach works well: route warnings to a Slack channel during business hours, and page the on-call engineer only for critical issues.

One critical gap: someone must own these metrics, and this should be explicit in the handover. If the system transfers to the customer's team without a designated metric owner and a regular review cadence, the best dashboard will sit unused within months. Monitoring typically fails for organizational reasons, not technical ones.

At Tenten, the SLA template is part of the launch deliverable. Engineers work with the customer from day one to establish thresholds, alert routing, and review frequency. The team stays involved for several weeks to calibrate settings manually. A polished demonstration is not the measure of success. Success is whether someone is still monitoring these metrics three months after launch and using them to make decisions.