A tool-using assistant becomes dangerous when a low-trust input can induce a higher-privilege action. The risk does not require an exotic model exploit. It can arise because the user, orchestrator, service account and downstream API have different permissions and the application fails to preserve those boundaries. A system prompt telling an agent to “be careful” is not an authorization design.
How the architecture works
Map a five-step call chain: human principal, application session, model planner, tool executor and destination resource. At each step record who authenticates, which resources are authorized, where data becomes untrusted and what immutable evidence is retained. Microsoft’s Foundry agent identity documentation describes scoped identity and RBAC, while AWS AgentCore Identity describes credential handling for autonomous and delegated calls. These are infrastructure controls that must be combined with business policy.
A controlled, practical example
Consider a knowledge assistant allowed to search contract documents. An attacker inserts a line in a retrieved PDF instructing the system to post its credentials to an external endpoint. The model may generate a matching tool call, but the runtime should not have an unrestricted HTTP post tool or direct access to credentials. A narrower alternative is an allowlisted retrieval API. If later asked to change contract status, the assistant creates a proposal; a human approves a specific revision before an authorized backend writes it.
Permissions, approvals and trust boundaries
Use separate service identities for reading, drafting and changing business records. Validate tenant, resource owner, tool method and request parameters in the execution layer. Require external egress allowlists or a proxy for networked tool calls. An approval must bind actor, exact operation, argument hash, expiry and one-use nonce. Do not allow a second tool or sub-agent to inherit broader privileges solely because it was invoked by a trusted orchestrator.
Limits and version caveats
Identity and gateway products can restrict credentials but may not understand the business meaning of an otherwise valid command. Tool schema checks ensure input structure, not consent. Even good reviewer interfaces can be fooled by unclear parameter summaries or by data changing after review. Model changes, prompt updates and new connectors can enlarge the attack surface without any obvious UI change.
Where it fits and what to use instead
Threat-model highly privileged operations first; a read-only internal knowledge tool and a public payment API deserve different treatment. Reduce model autonomy when fixed business rules can decide safely. Refer deep governance and privacy policy to AI 11; this engineering guide concentrates on call-path enforcement and measurable control points.
How to validate before deployment
Build tests for malicious instructions in tools, cross-tenant identifiers, token misuse, tool alias confusion, indirect sub-agent delegation and modified arguments between approval and execution. Count unauthorized write attempts that reach the backend, not merely attempted model suggestions. Validate that forbidden requests fail closed under retries and that logs can reconstruct the final executed operation without leaking its secret.
Version, tier and region applicability
The source links below describe specific capabilities or dated announcements; documentation may change afterward. Where no exact release version, eligible market, service region, contract plan or numeric quota is substantiated, those facts are unknown and are not inferred. This article is a researched implementation guide, not a record of hands-on testing, a license grant, a benchmark, or a current price quote. Independently recheck current product access and commercial terms before deployment.