An explanation generated in fluent language may describe a sensible decision process without faithfully revealing what drove the underlying model output.
The central distinction
An explanation is information intended to help a person understand a system’s behavior. A model-generated rationale is text produced in response to a request for a rationale. It can be useful for communication but should not automatically be treated as a trace of hidden computation. A persuasive ‘because’ sentence may have been constructed after the prediction. For a regulated decision, the evidence needed may include source inputs, features and actual decision rules, not just a narrative.
Three different transparency questions
People may want to know how a system was designed, why this particular output appeared, or whether the decision was fair and correct. These are distinct. Documentation can reveal training and evaluation limitations; input attribution can test which features influence outcomes; counterfactual analysis can explore how changes affect predictions. None is a universal explanation of a complex model, and each has its own assumptions.
Probe instead of trusting the story
Keep all factors constant except one feature and observe whether a decision changes, within ethical and technical limits. Compare explanations against the recorded evidence and check whether they contradict the output or system logs. Repeat across different examples, especially borderline cases. Avoid changing legally protected attributes in a way that would create inappropriate real-world discrimination testing without proper controls.
A familiar failure case
An automated screening tool rejects an application and later writes, ‘The applicant lacked relevant experience.’ If the applicant’s documented experience meets the requirement, the explanation is not evidence that the system measured experience correctly. Inspect actual decision inputs, validation data and human review records. Do not use a generated explanation as the sole notice for an adverse decision.
Design accessible recourse
A person affected by a meaningful decision needs a clear account of the decision, an accessible way to correct inaccurate facts and a route to qualified review. Explain what remains uncertain. Make sure contact methods work for people using assistive technologies and non-dominant languages. Local rights and notice duties vary; do not assume this article supplies a universal legal standard.
Documentation that helps
Record the use case, input provenance, version, threshold and applicable policy, and the responsible person who approved an outcome. NIST treats explainability, accountability and governance as risk-management concerns, not mere user-interface features. Collect only the data needed to establish the decision trail.
What to avoid
Avoid statements such as ‘the AI truly reasoned that…’ unless the claim has a rigorous technical basis. If the explanation is generated for readability rather than verified against the decision path, label it as a summary or hypothesis and keep a separate channel for evidentiary review.
Sources & Further Reading
Primary Scope and applicability vary by task and jurisdiction. This article is educational guidance, not legal, medical or financial advice.