A harmful AI result is not always a conventional software crash. It may be an unauthorized disclosure, fabricated guidance, discriminatory decision, unsafe action or misleading synthetic-media release.
Prepare before the first incident
Define what constitutes an incident, who can stop the system, how to preserve appropriate evidence, which teams investigate, and who may contact affected users. Keep an approved escalation path for security, privacy, legal, editorial and product staff. If a model-connected tool can take actions, ensure a safe-disable mechanism works without asking the model’s permission.
Triage impact and scope
Identify what happened, which users or records were affected, what data may have left the environment and which system components changed. Separate observed evidence from plausible hypotheses. A model producing a harmful sentence does not alone prove data theft; conversely, a tool log may show outbound exposure even when chat text looks normal.
Contain and recover
Revoke compromised credentials, freeze risky connectors, roll back model or prompt versions when appropriate, quarantine hostile content and notify accountable owners. Preserve logs lawfully and proportionately. Avoid a “fix” that deletes all evidence before the team can assess breach notifications or recurrence risk.
Write a useful postmortem
Include timeline, exact configuration and model alias if available, impact, root causes, failed barriers, effective mitigations, owners and test cases added to the regression suite. Explain whether the event resulted from hallucination, adversarial manipulation, flawed permissions or process failures; several causes can coexist.
Example: fabricated legal citations
A document assistant cites a non-existent judgment. The team pauses automated publication, checks related outputs, notifies affected editors, updates verification rules, adds a test for fabricated authorities and documents why human review did not catch the issue. Merely changing the system prompt is insufficient evidence of remediation.
Compliance caveat
Notification time limits and regulatory obligations are jurisdiction- and incident-specific. This article intentionally does not invent a universal reporting deadline. Use the applicable privacy, cybersecurity and sector rules to determine external notices.
Sources & Further Reading
Guidance varies by jurisdiction and application. This is general education, not legal, medical or other professional advice.