A glossy showcase can be persuasive without proving how reliably a model works. To evaluate AI video for production, a small team needs a repeatable rubric and honest sampling rather than an invented leaderboard. This article describes a proposed internal test design; no provider has been secretly tested and no comparative score is being reported.
Fix the task before picking a model
Define several representative tasks: a single object turning under fixed lighting, a person interacting with a prop, a product maintaining visible labels, a moving camera through a doorway and a two-speaker scene with audio. Each has different failure modes. The tested model/version, account plan and date must be recorded; a model that changed after the trial cannot be assumed to retain the same behavior.
Use the same authorized inputs whenever technically possible. If one model requires an image and another takes text-only, either stratify the results or label the comparison as non-equivalent. Equal output seconds or equal monetary cost are distinct experimental conditions. Do not claim a fair comparison when an expensive model receives many retries and a cheaper one receives only one attempt.
Define observable criteria
Score motion clarity, temporal stability, object permanence, identity consistency, framing, prompt adherence, audio synchronization and practical editability. Give reviewers concrete instructions. For example, a ‘temporal stability’ failure might be a lamp changing shape during a pan; an ‘identity’ failure might be a face or outfit changing across two linked shots. Also record safety or factual failures as binary disqualifiers rather than averaging them away.
Have at least two reviewers inspect clips at full speed and at native pixel dimensions without knowing the provider where feasible. Disagreements should be documented and resolved against the written brief, not against a preferred brand. Save the source prompt, randomization procedure, individual judgments, version metadata and rejection reasons. When reporting results, separate first-pass acceptance from best-of-many showcase output.
Cost and latency in the same protocol
Include actual charges, retry count, end-to-end elapsed time and edit minutes if measured. If those were not measured, report ‘not tested’ rather than a fabricated average. Vendor announcements about being faster, more accurate or higher ranked are not evidence of independent performance. A legally safe production option may be preferable to a visually impressive result with unclear rights.
Make the result actionable
The useful outcome is a decision: this class of shot is suitable for first-pass AI generation; that one needs a manual compositor; another is forbidden due to missing likeness consent. Preserve failures as examples for the storyboarding team and revise prompts only after the acceptance criteria are stable. Independent rankings and formal benchmarking fall under AI 09; this rubric helps AI 04 producers decide whether a clip meets their own brief.
Worked production example
An internal evaluation might contain ten fictional product shots, five camera-motion shots and five avatar-localization clips. Before generation, define pass/fail rules for the category: product geometry and labels, camera-path stability or phoneme alignment. Mask vendor identifiers for human reviewers where possible. Record first-attempt results separately from best-of-five retries and never publish a single unexplained average. If one model cannot accept the same inputs, either create a separate cohort or report that the comparison lacks equivalent controls. The protocol, not a prettier demonstration, determines whether the conclusions are credible.