An AI safety score without a test version, model snapshot or evaluation date can mislead even when the original result was accurate.
A benchmark is a package of decisions
Every score depends on the cases sampled, instructions given, outputs collected and labels assigned. Revise any of these and you may no longer be measuring the same thing. A benchmark version should identify the dataset, categories, grader, scoring definitions, sample size and allowed tools. A model version should identify the evaluated build or, if immutable IDs are unavailable, the access date and environment.
Why old screenshots mislead
A leaderboard image shared months after an evaluation may omit that the vendor updated the model, modified its default safety settings or retired the original endpoint. Another team may reproduce the prompt using a different decoding configuration and report a different score. Neither number can be compared fairly without understanding the protocol change. Do not reprint a rank as a current product fact unless both model and test states are known.
Keep benchmark history readable
Archive machine-readable results with date, revision, status and a description of changes. Do not overwrite an older score with a newer one under the same field name. If a benchmark changes hazards or grading, create a distinct metric ID and explain discontinuities rather than charting an apparently smooth improvement.
Separate blocking from informational results
A passing benchmark may support an internal release gate only for scenarios the benchmark covers. Real deployments add retrieval, tools, multilingual inputs and user behavior. If tests do not cover those features, mark the missing areas ‘not evaluated’. Avoid treating unmeasured categories as zero risk. Use additional adversarial and incident-informed tests where the application permits.
Example: new safety grader
Suppose a company re-scores last month’s generated answers with a stricter judge. The percentage of violating outputs rises even though the model did not change. An honest dashboard annotates the grader revision and avoids presenting the increase as proof of model deterioration. Re-score a bridge sample using both graders if comparisons are needed.
Outside review
Stanford HELM emphasizes transparency of prompts and evaluation configurations. MLCommons publishes boundaries around its safety assessments. These practices make scrutiny possible, but neither automatically establishes the validity of your local tasks. Preserve first-party releases and distinguish them from independent retests.
Minimum metadata
Each result should carry benchmark ID/version, task or hazard, locale, prompt configuration, scoring rubric and grader, model identifier or access date, sample count, uncertainty where applicable, and owner. Unknown values should stay unknown rather than be filled from a later release note.
Sources & Further Reading
- AILuminate benchmarks and methodology — MLCommons.
- Holistic Evaluation of Language Models — Stanford CRFM.
- NIST AI RMF Core: Govern Map Measure Manage — NIST.
Primary Scope and applicability vary by task and jurisdiction. This article is educational guidance, not legal, medical or financial advice.