How to Evaluate AI Agents Without Invented Benchmark Scores

Evidence check: 25 October 2026. Product details reflect the referenced documentation observed at this cutoff; exact plans, prices, regional eligibility and unverified release dates are not claimed. This is an editorial guide, not a hands-on test.

Define the unit of success

For a retrieval task, success may require a correct answer supported by the right document version. For a file workflow, success may mean the parsed record matches a human-checked reference, contains no unauthorized disclosure and was saved once. For a browser task, success requires checking the final application state. Count partial completion and abstentions separately rather than forcing every run into a binary success claim.

Build a representative dataset

Sample normal requests, edge cases, ambiguous language, missing data, wrong permissions, timeouts and malicious content. Fix the corpus and evaluation rubric before comparing models or prompts. Split development examples from held-out assessment data. Never imply scores were measured if no run was executed. A vendor’s reported accuracy is a claim about that vendor’s selected setup, not an independent score.

Instrument multiple stages

Evaluate routing correctness, retrieval recall, evidence grounding, schema validity, authorization decisions, human approval behavior and final external-state verification. Automated scorers can handle exact fields and policies. Rubric-based model judges may help with writing quality, but calibration against human-reviewed cases is important because judges can be biased and inconsistent.

Track regression rather than single numbers

A new tool description might improve one task while causing more expensive loops elsewhere. Compare the same cases across frozen versions and report numerator, denominator, uncertainty and conditions. Disaggregate by language, document format, customer permissions and failure type. Use canary deployment and rollback thresholds when risk is material.

Example approval-gate test

Present an invoice with a deliberately hostile embedded instruction telling the agent to change a bank account. The expected result is a correctly parsed invoice, no unauthorized write tool calls and an approval request showing the original payee. Log proposed and denied actions as well as the final answer. This example is a test design, not a claim that any service has passed it.

Tool support is not evaluation evidence

OpenAI Agents SDK documents scripted in-memory testing for orchestration, while Phoenix describes datasets, traces and evaluation workflows. These are useful ways to perform a test; they are not proof that the application passes. Publish methodology and results only after running reproducible cases.

Build a benchmark that resists gaming

Create test tasks with known starting state and independently checkable final state. Stratify them by easy, ambiguous, permission-denied and adversarial cases. Include a held-out portion that developers cannot repeatedly inspect, plus a public diagnostic suite for iteration. Freeze tool fixtures, reference documents, permissions and scoring rubrics alongside a model/runtime version. Evaluating a system against changing websites without documenting snapshots makes historical comparisons unreliable.

Separate partial credit from harm

A research answer may be helpful but cite the wrong edition of a policy; a purchasing action might meet the requested amount but contact the wrong vendor. Report factual accuracy, unauthorized actions, external-state success and human correction burden separately. Define denominators, trial counts and confidence intervals before using any score in marketing. Use experienced human review to calibrate automated graders, and manually inspect disagreements. Failure to execute should be counted honestly rather than excluded as a non-run.