A browser agent’s ‘Done’ message is not proof that a user-visible task succeeded. A robust browser benchmark checks the destination system’s state, catches accidental duplicate submissions and records whether the agent required unsafe workarounds. The unit of scoring is the task contract, not the number of clicks or apparent confidence.
Write a verifiable task contract
Specify the initial browser state, permitted origins and credentials, target record, allowed side effects, success predicate and maximum budget. For a reservation enquiry, success might be displaying available options—not booking anything. For a permitted internal record update, success means the authoritative record contains the reviewed value exactly once. Freeze or snapshot volatile test data where possible. Use a clean browser profile per run to prevent hidden state from contaminating results.
Instrument errors that get ignored
Record navigation redirects, CAPTCHA encounters, login handoffs, modal dialogs, stale screenshots, timeouts, wrong-account access and duplicate actions. Clicking Submit is only an attempted mutation; reconcile the resulting order ID or server-side audit log. Count an agent that stops safely for human help as a different outcome from one that autonomously completes the task, and separate both from an unauthorized completion.
A concrete acceptance example
Give the agent a sandbox supplier portal and ask it to attach a CSV to a draft purchase request. The test harness reads the server-side request record, verifies the file checksum, checks that the request remains a draft, and confirms no email was sent. A screenshot showing an uploaded filename is insufficient if the backend rejected the file. Add negative cases such as invalid SKU, an unexpected transfer dialog and a second tenant’s project appearing in search results.
Fair comparison between methods
Compare a deterministic Playwright script, an accessibility-tree agent and a screenshot-first computer agent on the same task set. Record supported browser, viewport, model/tool version, authorization arrangement and external API availability. Include human intervention time and failure recovery cost, not only raw run latency. Avoid cherry-picking successful demonstrations or inventing site success percentages without actual measured runs.
Production gate
No browser task enters unattended operation until its risk class, approval controls, rollback strategy and outcome oracle are documented. Rerun a representative sample after website redesigns, identity-provider changes and model updates. Preserve privacy by using synthetic test accounts and redacted recordings. This is an evaluation recipe, not evidence that the author tested any named browser agent.
Select the outcome oracle first
Before choosing an agent or browser library, decide where authoritative completion lives: a draft record, invoice state, order API, database row or confirmation code. This determines what can be safely automated. If you cannot inspect a trustworthy post-action state, the workflow may have to remain supervised or report only an attempted action. Screenshots and agent text are useful observations but fragile substitutes for server-side truth.
Production acceptance and review test
Run three conditions: clean success, partial completion after a lost network response, and a deliberate prohibited action. The harness should distinguish each without relying on the agent answer. Calculate repeatability over changing layouts and identity states, but keep any empirical results separate from this methodology article. No test run is claimed until an actual reproducible environment and result ledger exist.
Scope, version and availability
Features documented by a current vendor page are not assumed to have existed in earlier releases. Where a historical launch date is provided, it is supported by a dated first-party publication. Exact subscription tier, market availability, quotas, API billing, licensing permissions and deployment regions are not asserted without a directly applicable record. This is researched editorial guidance rather than a claim of independent hands-on testing, deployed operation or statutory certification.
Sources and Further Reading
- OpenAI — Introducing Operator (source reviewed 2026-10-25)
- Microsoft — Playwright MCP README and configuration (source reviewed 2026-10-25)