Not every AI attack looks like a malicious chat prompt. Some adversaries try to contaminate what a system learns; others change the inputs it sees while it is operating.
Poisoning changes the learning or reference environment
An attacker may manipulate training examples, labels, fine-tuning material or documents later retrieved as evidence. The result can be a targeted wrong answer, degraded accuracy or a hidden behavioral preference. The attacker’s practical access matters: controlling an indexed web page is different from modifying the original model’s training dataset.
Evasion targets live behavior
An evasion attack tries to induce an error on an otherwise deployed system by crafting an input that changes how it responds. In predictive machine learning, this might involve carefully modified observations. In generative systems, harmful instructions may be blended into otherwise legitimate content. The categories overlap with prompt injection but are not identical.
Establish realistic controls
Control data ingestion, document ownership, update rights and labeling quality. Maintain protected holdout evaluations so that poisoned samples cannot silently become both training and tests. Compare behavior after updates and investigate unexplained changes by topic, subgroup or source. Test against adversarial cases using isolated fixtures, and define rollback authority.
Understand detection limits
A suspicious outlier is not definitive proof of an attack, and attacks need not be obvious outliers. A benchmark against known adversarial examples cannot promise immunity to novel manipulations. Protect upstream collection and access controls instead of relying only on a classifier to spot malicious content.
Example
A support retrieval index accepts community posts without provenance. Attackers repeatedly insert a fabricated return policy. The assistant begins quoting that policy as official. An effective defense distinguishes approved policy documents from community discussion, pins effective versions and routes uncertain cases to verified staff rather than trusting whichever paragraph ranks first.
Source context
NIST AI 100-2e2025 defines the lifecycle and attacker-capability terminology for these risks; its publication does not mean a specific deployed model has been shown vulnerable without testing.
Sources & Further Reading
Guidance varies by jurisdiction and application. This is general education, not legal, medical or other professional advice.