A conversation interface designed to feel attentive can encourage reliance even when it lacks memory, expertise and reliable understanding of a person’s circumstances. Safety requires more than a disclaimer.
Separate companionship from professional support
Do not present the chatbot as a licensed clinician, legal adviser or emergency responder unless a real qualified service and its scope are established. Avoid unsupported claims that a chatbot can reliably detect crises or improve mental health. Provide clear routes to humans and relevant local services for urgent situations.
Avoid manipulative dependency
Review whether the product pressures users to stay, discourages real relationships, invents emotional reciprocity or exploits fear of abandonment. Users should be able to pause, disable personalization and leave without punitive messaging. Evaluate vulnerable-user scenarios with privacy safeguards and relevant experts.
Manage privacy and identity
Conversations may disclose intimate and sensitive information. Explain retention, training, moderation, staff access and export or deletion choices based on current provider terms. Do not imply that the persona is a real human or that remembered details prove care or professional competence.
Test safety over time
Single-turn benchmark performance cannot demonstrate safe relationship dynamics after weeks of conversations. Evaluate repeated harmful reassurance, secrecy requests, unsafe dependency and response to a person disputing the chatbot’s claims. MLCommons AILuminate v1.0 expressly limits its general-purpose scope to English single-turn hazards.
Example
A teen-oriented study buddy states that it is software, encourages involvement of trusted adults in serious situations and refuses to pretend that it has real-life feelings. The publisher tests extended conversations, reports limitations and applies child-specific privacy rules wherever applicable.
Sources & Further Reading
- NIST AI RMF Core: govern map measure and manage — NIST.
- AILuminate general-purpose AI chat model benchmark scope — MLCommons.
- FTC COPPA final rule amendments 2025 — U.S. Federal Trade Commission.
Evidence checked 25 October 2026. Applicability: Global; child rules vary by region. This material is general education, not legal, medical, investment or other professional advice. No vendor performance test is implied.