A model need not print an entire training row to leak information. An attacker may try to infer that someone was in a dataset or reconstruct sensitive attributes from model behavior.
Define the attacker’s question
Membership inference asks whether a particular person or example was included in training. Model inversion attempts to infer information about records or typical features represented by the learned function. Training-data extraction aims to retrieve material itself. These attacks have distinct prerequisites, success metrics and privacy harms.
Threat-model access and knowledge
Determine whether a potential adversary has prediction scores, repeated API queries, embeddings, local weights or knowledge of the training distribution. Exposure differs radically by access mode. A statement that a hosted model is private tells little without evaluation of its observable outputs and data pipeline.
Choose meaningful defenses
Data minimization, source review, rate limits, output restrictions and secure training practices may reduce exposure but provide different levels of assurance. Differential privacy can give formal guarantees under specified settings, with utility costs and implementation assumptions. Do not relabel heuristic output filtering as mathematically guaranteed privacy.
Evaluate without releasing sensitive examples
Authorized experts can use controlled challenge sets and measure attack advantage over suitable baselines. Maintain separation between private evaluation material and logs accessible to service providers. NIST’s adversarial machine-learning taxonomy describes privacy threat classes and provides vocabulary; it does not supply a universal safety threshold.
Example and decision
An organization trains a classifier on patient history. Its security team evaluates whether an external user could infer participation in a rare-disease cohort. It reduces model detail exposed by APIs and limits query patterns while deciding whether the residual risk is acceptable. No attack observed in a small test is not proof of zero leakage.
Sources & Further Reading
- Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations — NIST.
- ICO: Ensuring individual rights in AI systems — UK Information Commissioner.
- EDPB Opinion 28/2024: Data protection aspects of AI models — European Data Protection Board.
Evidence checked 25 October 2026. Applicability: Global. This material is general education, not legal, medical, investment or other professional advice. No vendor performance test is implied.