An AI answer may reproduce a private passage because the model memorized it during training—or because a connected search tool supplied it five seconds earlier. The incident response is different.
Trace what the model could access
Capture the model release, prompt, user authorization, retrieval results and tools invoked. If an answer comes from a retrieved document, check whether the retrieval service enforced the original user’s permissions. If no source was retrieved, investigate pretraining, fine-tuning data, cache and prompt history paths.
Use controlled experiments
In a staging environment, disable retrieval and repeat carefully constructed queries with no actual secrets. Test canary passages known only to training or retrieval systems. Do not assume that repeated reproduction conclusively proves training origin: system prompts, caches and hidden middleware may provide data.
Assign the right remedy
For an access-control lapse, fix indexing, document permissions and query-time filtering; rotate disclosed secrets. For training memorization, evaluate source removal, model retraining, targeted mitigations and available rights processes. Restrict outputs while investigation proceeds. A generic safety filter can miss transformed or partial disclosures.
Know the legal and technical limits
The UK ICO notes that some personal data may be deliberately or accidentally contained in a model and that rights requests can become difficult. Whether the record is personal data and which rights apply need specific analysis. Security fixes do not substitute for identifying lawful collection and training purposes.
Example
An assistant reveals part of a confidential employee policy draft. Engineers find the draft was indexed with a public ACL even though the original document was private. Correcting retrieval authorization and cache invalidation is the primary response, not retraining an unrelated foundation model.
Sources & Further Reading
- ICO: Ensuring individual rights in AI systems — UK Information Commissioner.
- OWASP Top 10 for LLM and Generative AI Applications 2025 — OWASP.
- Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations — NIST.
Evidence checked 25 October 2026. Applicability: Global. This material is general education, not legal, medical, investment or other professional advice. No vendor performance test is implied.