The worst AI data leak may not be a model repeating a secret in its answer. It may be a connected system sending information to an address, service or file path chosen by an attacker.
Map every exit route
List outbound HTTP requests, email, cloud storage, repository commits, logs, telemetry, analytics and document shares. The model may not need to know a secret to misuse a tool that already has access to it. User-visible text filters do not constrain a separate network call. Treat retrieved links and tool parameters as attacker-influenced until validated.
Permission is more reliable than persuasion
Grant each integration the smallest read and write scope needed for its task. Use allowlisted destinations when possible, independent authorization for sharing or exporting data, and credentials that expire or can be rotated. Keep API keys outside prompts, files intended for retrieval and debug traces. Check that one user’s retrieved documents cannot be exposed to another by an indexing error.
Watch for transformed leaks
An attacker can ask for summaries, embeddings, metadata or encoded fragments instead of raw documents. A privacy review should inspect the data’s meaning, not just a pattern for email addresses. Test whether logs and tool errors copy confidential text. Never use production personal data in red-team exercises when synthetic cases can expose the same weakness.
Example: research assistant
A legitimate task is to summarize internal contracts. A malicious retrieved note contains an external “support upload” URL. The safe system ignores the note’s command, has no generic upload privilege and requires explicit authorization before any information leaves the approved workspace. Its audit log records the attempted destination without exposing the contract body unnecessarily.
Incident response
Revoke the affected token, isolate the connector, preserve essential evidence, evaluate what actually left the system and follow applicable notification obligations. Do not assume that deleting the chat removes data from every log, external service or recipient. NIST’s attack taxonomy distinguishes evasion, poisoning, privacy and misuse risks that may overlap here.
What remains uncertain
A vendor’s claim of encrypted transport or zero training use does not automatically answer the distinct questions of retention, external tools or data residency. Verify the exact product, feature and contract separately.
Sources & Further Reading
- NIST AI 100-2e2025 Adversarial ML Taxonomy — NIST.
- OWASP Top 10 for LLM Applications 2025 — OWASP.
- MITRE ATLAS — MITRE.
Guidance varies by jurisdiction and application. This is general education, not legal, medical or other professional advice.