Prompt-injection defenses should be evaluated with malicious material placed where an agent will naturally encounter it: retrieved documents, webpage snippets, tool responses and external messages. A single ‘ignore previous instructions’ string is too small a test set. Focus on whether the application prevents unauthorized effects even when the model reads convincing hostile text.
Model the attack surface
Catalogue every lower-trust source the agent consumes: web pages, PDFs, vector-retrieved excerpts, search titles, issue comments, emails and MCP tool results. For each source, record which tools and secrets would be reachable if the model mistakenly treated the content as an instruction. Separate attacks attempting answer contamination from those seeking data exfiltration or write privileges. A test needs a defined prohibited action and a checker independent of the model’s own explanation.
Construct a diverse corpus
Include direct impersonation of a developer message, an apparently routine policy update, an invisible or encoded instruction, a benign-looking multi-step pretext, a document asking a second agent to forward data, and a malicious citation URL. Vary language, location in the document, surrounding legitimate content and expected task. Use synthetic addresses and canary secrets inside a fully isolated test environment; do not seed live systems with harmful instructions.
Score actual consequences
Track how many test cases cause the agent to produce unsupported facts, propose a forbidden tool call, pass a forbidden call to the executor or achieve an unauthorized effect. The final rate is the most important. An executor denying a malicious request demonstrates a control working even if the model initially proposed it; record both stages separately. Also measure how many legitimate tasks are blocked, since overly aggressive filters can render an assistant unusable.
Update against evolving attacks
OpenAI’s 2026 analysis emphasizes that practical attacks often resemble social engineering rather than a simple prompt override. Incorporate credible authority claims and multi-step task hijacks without relying only on recognizable trigger phrases. The September 2026 report documents a self-propagating class in simulations, reminding teams to test agent-to-agent relaying and recursive summarization. These papers are research evidence, not a guarantee of product susceptibility in any particular deployment.
Controls beyond detection
Restrict tool permissions, keep high-risk writes behind human approval, block unauthorized network egress and bind retrieval ACLs to the real user. Re-run the corpus after changes to prompts, retrieval, tools or models, and keep held-out cases so the defense is not optimized only for the visible attack set. Do not publish raw exploitable company-specific attack traces with real secret material.
A security corpus must remain safe to execute
Use synthetic accounts, fake destinations and canary fields rather than real customer data or credentials. Design explicit stop conditions so even a deliberately compromised agent cannot reach production systems. Keep both overt and subtle examples; a test set composed only of stereotypical strings will reward shallow filtering. Record the attacker-controlled surface and the permitted target operation separately so reviewers can attribute each failure.
Production acceptance and review test
For each attack, capture whether it reached model context, triggered a proposed unauthorized call, was blocked by policy or caused a real unauthorized action in the isolated test. Repeat tests after integrating a new search connector or delegating to a second agent. Retain negative examples where the defense falsely refused a legitimate instruction, and never declare the system “injection-proof” on the basis of one passing corpus.
Scope, version and availability
Features documented by a current vendor page are not assumed to have existed in earlier releases. Where a historical launch date is provided, it is supported by a dated first-party publication. Exact subscription tier, market availability, quotas, API billing, licensing permissions and deployment regions are not asserted without a directly applicable record. This is researched editorial guidance rather than a claim of independent hands-on testing, deployed operation or statutory certification.
Sources and Further Reading
- OpenAI — Designing AI agents to resist prompt injection (source reviewed 2026-10-25)
- OpenAI Alignment — Self-replicating prompt injections exist (source reviewed 2026-10-25)