A model can propose a tool argument, but only trusted application code should decide whether that argument is valid and authorized.
Text is not permission
AI applications may send model-generated arguments to search tools, file operations, databases, payment systems or messaging APIs. The model’s output can be malformed, based on a misunderstanding or influenced by an attacker-controlled document. Treat it like untrusted input. JSON validity alone does not mean that a command is safe, that the action fits the user’s request or that a model may exercise the underlying permission.
Validate at the boundary
Define strict schemas, allowed values, object identifiers, numeric limits and acceptable destinations. Resolve resource identifiers against server-side permissions before execution. Do not allow the model to choose an arbitrary endpoint, tenant or file path just because it has generated syntactically correct arguments. Apply least-privilege credentials, scope-limited tokens and separation of read from write privileges.
Check purpose as well as shape
A request to ‘show my invoice’ does not authorize sending it to another email address. After schema validation, compare the proposed action with the authenticated user’s intent and any required approval. For irreversible operations, display a clear confirmation identifying the target and consequence. Do not obtain apparent consent through text supplied by a retrieved web page or email.
Defend against repeated actions
Retries are useful for unreliable networks but hazardous for payments, deletions and messages. Use idempotency keys where supported, detect duplicates and record outcome before retrying. Handle partial failures explicitly. A tool timeout does not prove that an action failed; blindly repeating it can produce a second transaction. Apply rate limits and cost controls independent of the model’s preferences.
Failure example
A calendar assistant receives malicious meeting notes containing a fake instruction to invite an outside address. A robust system treats the notes as content, checks whether the user actually requested invitations and requires approval with recipient details. A model-generated explanation that the invitation is routine provides no authorization.
Testing and monitoring
Fuzz the validator with invalid types, unexpected Unicode, oversized strings, cross-tenant IDs and malformed URLs. Verify that rejected calls produce no side effects and that logs do not expose secrets. Monitor newly introduced tools and configuration changes. Tests must run in an authorized controlled environment; do not target third-party services without permission.
Limits and handoff
This article describes the security principle, not a particular orchestration framework. AI 07 owns operational agent workflow design. NIST and OWASP provide broader risk guidance, but each integration must enforce its own application-specific permissions and jurisdictional requirements.
Sources & Further Reading
Primary Scope and applicability vary by task and jurisdiction. This article is educational guidance, not legal, medical or financial advice.