Evaluation Drift

AI quality can change when inputs, users, reference material, models, prompts or tools change. Infrastructure uptime and latency do not describe every change in task quality. Establish an accepted-task baseline and compare later results on the dimensions that matter to the workflow.

Anthropic’s 9 January 2026 guide to agent evaluations discusses code-based, model-based and human graders, calibration and complementary monitoring. This paper applies those distinctions to an evaluation plan: representative cases, risk-based sampling, explicit alert decisions and an owner for investigation. No fixed judge-agreement percentage, sample count or monitoring cadence establishes production reliability for every system.