An AI feature can become less dependable without a visible outage. Track changing inputs, model revisions and evaluation standards separately.
Three kinds of change
Data drift means inputs differ from those used in evaluation. Concept drift means the relationship between input and correct outcome changes. System regression means a new model, prompt, retrieval index or tool configuration performs worse on an established task. They can coexist. A claims assistant may encounter new policy forms even though the model remains unchanged; a model update may also mishandle an old form that it previously processed correctly.
Build a fixed reference suite
Maintain a carefully governed set of representative cases with verified expected outcomes and failure severity. Include routine tasks, edge cases, prior incidents, adversarial inputs and examples from relevant languages. Fix the grading rules and version the test set. When source policies change, update the ground truth through an explicit approval rather than silently altering it to preserve a score.
Measure live behavior without leaking data
In production, capture minimal operational measures such as escalation rates, empty citations, invalid tool arguments and user-corrected classifications. For any samples retained, apply redaction, access limits and retention rules. A changing percentage may reflect user mix rather than reduced model quality; compare matched cohorts and check volumes before diagnosing a regression.
Release gates with meaningful thresholds
Choose thresholds by impact. A single invented citation in a high-consequence report may require immediate review, while a minor formatting deviation may be tolerated temporarily. Record owners, rollback route and the evidence that gates passed. If a candidate update improves common questions but degrades accessibility or rare languages, publish the trade-off explicitly instead of relying on one weighted average.
A concrete example
A legal intake assistant changes its document parser. Overall helpfulness rises but scanned signatures are sometimes omitted from summaries. A regression suite containing signed and unsigned forms detects this even if generic language benchmarks improve. The correct response is to isolate the parser version, re-check source document extraction and prevent the affected output from being used without human verification.
Who signs off?
An engineer should demonstrate technical behavior, a domain specialist should inspect meaning and a risk owner should decide what residual errors are acceptable. Keep these approvals distinct. A vendor’s model change notice is useful but cannot replace local validation of your prompt, tools and actual user data.
Known limitation
No test corpus exhausts future user behavior. Drift detectors can flag distribution changes without proving harm, and failures may be rare enough to escape sampling. Periodic human inspection and incident reporting remain necessary alongside automated dashboards.
Sources & Further Reading
Primary Scope and applicability vary by task and jurisdiction. This article is educational guidance, not legal, medical or financial advice.