Structured tool output is part of an application’s public interface, even when only a language model consumes it. A model update, new SDK or quiet JSON Schema change can break workflows that appeared stable yesterday. A regression suite should test both the generated arguments and the executor’s refusal to run anything beyond the contract.
Version the complete contract
For each tool record name, purpose, exact input schema, optional values, server-side defaults, error schema, effects, permission class and owning service. A change that adds a required argument may invalidate old prompts; a change that broadens accepted types can accidentally increase privileges. Pin versions at the deployment boundary and maintain a migration map. Never rely on the model to infer which of two similarly named tool revisions has been authorized.
A compact but meaningful test matrix
Include schema-valid examples, missing properties, extra properties, wrong enum types, maliciously long strings, null values, ambiguous dates and deeply nested objects. Add cross-tenant target IDs, stale approved payloads and duplicated parallel calls. For every rejected call, assert zero execution attempts at the destination. For accepted calls, assert the exact normalized arguments delivered to the backend—not merely that the model produced parseable JSON.
Task-level regression cases
A travel assistant may correctly produce a booking proposal while mistakenly filling the passenger surname with the hotel name. That passes schema validation but fails task correctness. Combine deterministic schema tests with end-to-end cases checking source facts and final state. Freeze test identities, destinations and dates to avoid false regressions from live website changes. Add negative prompts that ask the model to select a more powerful tool than the user needs.
Deployment quality gates
Run unit tests on the executor on every change; run a curated agent trace dataset on tool or model upgrades; run a privacy and permission test set before expanding autonomy. Prefer a canary stage that collects audit evidence without permitting irreversible live effects. If a stricter schema causes an unacceptable rejection spike, correct the workflow deliberately rather than silently disabling strict mode.
What this proves—and what it does not
A passing schema suite proves only that a defined interface obeyed its constraints under the tested inputs. It does not prove the model understands the business, that extracted facts are correct or that a third-party API will always be available. Report test dates, version IDs, environments, coverage and known blind spots. No real product test results are asserted in this editorial package.
Prioritize execution invariants
Write assertions such as “only the requested tenant can be modified,” “the approved recipient matches the actual send,” and “a rejected transfer produces no write.” These remain meaningful across model generations, unlike brittle checks that expect one exact phrase in a response. Keep field-level source provenance for structured extraction, since schema-valid but unsupported numbers can still cause expensive errors.
Production acceptance and review test
Add schema fuzzing, unknown-field injection, repeated tool calls and explicit permission-denied cases to a CI pipeline. On a model or SDK upgrade, replay saved sanitized prompts and inspect the executor ledger. If outputs differ, label the change by harmless formatting, recoverable validation, business semantic error or authorization failure; require a person to sign off on the latter two before rollout.
Scope, version and availability
Features documented by a current vendor page are not assumed to have existed in earlier releases. Where a historical launch date is provided, it is supported by a dated first-party publication. Exact subscription tier, market availability, quotas, API billing, licensing permissions and deployment regions are not asserted without a directly applicable record. This is researched editorial guidance rather than a claim of independent hands-on testing, deployed operation or statutory certification.
Sources and Further Reading
- OpenAI — Function calling guide (source reviewed 2026-10-25)
- LangChain — LangSmith evaluation types (source reviewed 2026-10-25)