An app recommends ordering forty units. The merchant asks why. The app produces a fluent paragraph about demand trends, but the paragraph does not show which source rows, date range or assumptions produced the recommendation. The explanation sounds complete while leaving the decision almost as opaque as before.
A useful explanation gives the customer evidence, relevant assumptions, uncertainty and a way to challenge or correct the result. It does not claim privileged access to a model’s true internal reasoning. The practical goal is an informed decision about this output and its consequences.
This chapter of The AI Software Factory owns the interface between evidence and customer judgment. Trust in software addresses credible promises and recourse; evaluation suites address system acceptance. Here the question is what a user can inspect and do when facing a particular AI result.
Begin with the decision the explanation supports
Not every explanation needs the same detail. A merchant deciding whether to reorder needs source quantities, dates and constraints. A user reviewing a document classification may need the relevant passage and competing categories. A manager approving an external write needs the proposed change and its effect.
Identify the user’s next decision before choosing an explanation format. “Why did the model say this?” can invite a long technical discussion that does not help the task. “What should I verify before accepting this reorder?” gives the interface a concrete purpose.
The decision also defines the required depth. A reversible label suggestion may need a brief source preview. A consequential transaction may need a complete change summary and explicit current authorization. Match evidence to the consequence rather than making every result equally elaborate.
Ask users to demonstrate how they would evaluate a result. Their actions reveal which information matters. A survey asking whether an explanation seems clear can miss a serious misunderstanding about what the app actually knows.
Show provenance before persuasive narrative
A source reference should connect the claim to the actual evidence used. For a proposed merchant report, show the relevant item identifier, source date range and the rows or calculations supporting the exception.
A citation to a general help page does not support a customer-specific quantity. A source link should also resolve to material the intended reviewer is allowed to inspect. Do not expose another account’s evidence simply to make the interface look transparent.
Preserve provenance through transformations. If the app normalizes names, groups rows or excludes invalid records, show enough context to explain the resulting quantity. Otherwise, the source may exist while the user cannot reproduce the relationship.
Separate evidence availability from evidence quality. A stale report with a perfect link is still stale. A document containing an unsupported assertion does not become trustworthy because the app cites it accurately. The interface should help users evaluate both origin and relevance.
Distinguish observations, calculations and inferences
These three kinds of statements deserve different representations. An observation records what the source contains. A calculation applies a defined operation. An inference interprets patterns under assumptions.
For a hypothetical inventory app, “Six units sold in the submitted period” is an observation if the source supports it. “Four weeks of stock at that observed rate” is a calculation. “Demand will rise next month” is an inference that may depend on information the app does not have.
Label the distinction without burying users in terminology. A small source panel, a visible formula and an assumptions section can be sufficient. The point is to let the merchant question the right part rather than treat the entire result as one indivisible AI answer.
This also improves correction. If the observation is wrong, fix source interpretation. If the formula is wrong, fix deterministic calculation. If the inference misses context, record the omitted constraint. The response should match the kind of error.
Show what the system did not know
Missing information can be more useful than a generic confidence label. A reorder analysis may lack supplier lead times, planned promotions or committed incoming stock. State those gaps when they materially affect the decision.
Connect each gap to a next step. “Supplier lead time unavailable; review before accepting quantity” is actionable. “AI may make mistakes” is broadly true but does not identify what this customer should inspect.
Do not list every possible missing fact for every output. That creates a warning wall that people learn to ignore. Identify the omissions relevant to the current result and the promised scope.
If required information is absent, the correct result may be a request for input rather than a recommendation. An explainable refusal to proceed can serve the customer better than a speculative answer wrapped in disclaimers.
Use uncertainty only when its meaning is established
A numeric confidence score can look precise even when it is not calibrated for the task. Do not present a model-generated number as a reliable probability merely because it is between zero and one.
If the product uses confidence estimates, define how they are produced, what they refer to and how they were checked. A score about classification differs from confidence in the eventual business outcome. The interface should not blur those targets.
Qualitative uncertainty can be useful when tied to evidence: conflicting source records, missing fields or an unfamiliar input pattern. The user can then understand why review is needed and what information might resolve it.
The OpenAI evaluation guidance discusses calibrating automated judgments with human review and notes biases in model graders. A model judge’s approval is therefore evidence to evaluate within scope, not an independent guarantee that the customer’s answer is correct.
Keep the explanation tied to the actual result
An explanation generated separately can accidentally describe a different version of the output. If the underlying recommendation changes, the source summary and assumptions should update with it.
Bind the explanation to the relevant output identity and source version. A customer reviewing yesterday’s report should see yesterday’s evidence, with a clear indication if newer information exists. Silently replacing the evidence can make the old decision impossible to reconstruct.
For external actions, the explanation should accompany the exact proposed payload. A general justification cannot authorize an amount or recipient that changes afterward. Safe writes develops that boundary.
When a correction changes the result, preserve an understandable history. The customer should be able to distinguish the original, the correction and the current accepted version. That supports accountability without forcing them to read internal implementation logs.
Layer information around the user’s task
Start with the result and the most consequential evidence. Offer deeper detail where it helps inspect a specific issue. A full audit trail can be available without making every ordinary interaction read like an incident report.
For a proposed report, the top layer might show the flagged item, source period, principal reason and missing constraint. A deeper layer could show source rows, normalization and calculation details. A technical export can support a specialized reviewer.
Avoid hiding material limits behind a collapsed panel while placing persuasive language in the visible layer. The first layer should contain what changes the immediate decision. Optional detail should add inspection depth, not reveal a contradictory promise.
Keep the navigation accessible and comprehensible. A reviewer should be able to find supporting evidence without precision gestures or ambiguous controls. The explanation is only useful if the intended user can operate it.
Give correction a structured path
A “thumbs down” button records dissatisfaction but may not reveal the error. Offer correction categories that match the workflow: wrong source mapping, missing business constraint, calculation issue or unsuitable recommendation.
Let the user provide the relevant correction without requesting unnecessary sensitive material. A field for lead time may be more useful than an unrestricted text box inviting customers to paste entire private documents.
Explain what the correction changes. Does it update only this result, the account’s future settings or a reviewed shared rule? Do not imply that every feedback submission immediately retrains the system or changes global behavior.
A correction can require review before reuse. User-provided information may itself be mistaken, outdated or outside the account’s authority. Preserve the provenance and scope of corrections rather than treating all feedback as trusted policy.
Make override legitimate and visible
A customer may have information the app lacks and reasonably reject its recommendation. The interface should permit that decision without treating the user as an obstacle to automation.
Record the accepted alternative and relevant reason when useful. For a hypothetical merchant app, the merchant may choose a smaller order because of cash constraints. That does not prove the model’s demand estimate was wrong; it shows a different decision objective.
An override should not erase the original evidence. Keeping the distinction helps future review identify whether the issue was data, prediction or a legitimate business tradeoff.
Authority still applies. A user allowed to inspect a recommendation may not be allowed to publish a change. The override interface should respect account roles and current payload approval rather than create a shortcut around the permission model.
Test whether explanations improve decisions
A useful explanation test asks people to complete the actual review task. Can they identify stale input, locate the supporting row, recognize an omitted constraint and choose the appropriate next action?
Compare with a defined baseline, such as the current report. Keep result difficulty comparable and preserve cases that challenge the explanation. A fluent explanation that makes users accept a wrong result faster is a failure, even if they rate it as clear.
Measure both correct acceptance and correct rejection. If every sample is accurate, the study cannot reveal whether the explanation helps detect mistakes. Include a known inconsistency and an unsupported case, clearly separated from real customer operations.
State the scope of any observed result. A small usability review can reveal confusion in the tested interface. It does not establish universal comprehension or safety across all users, languages and circumstances.
Do not expose secrets in the name of transparency
An explanation may reveal private data, system instructions, credentials or internal security details if it indiscriminately reproduces the material behind an output. Transparency needs an account and consequence boundary.
Show evidence the user is authorized to inspect and redact unnecessary private content. A support agent may need a job identity and error category rather than the customer’s full report. A teammate may need the decision summary without unrelated financial rows.
Privacy design traces data handling. Explanation features should participate in that lifecycle, including retention, export and deletion. A report panel is another data surface, not a special exception to the policy.
The product can explain relevant limits without disclosing hidden instructions or exploitable implementation detail. Users need to understand how to evaluate the result; they do not need every internal string the system encountered.
Explain deterministic constraints honestly
Some limits come from ordinary software rules rather than model uncertainty. A supported format, maximum job size or missing required identifier can be described directly. Do not attribute every restriction to mysterious AI judgment.
A clear validation message helps customers correct a problem. It also distinguishes a product boundary from an error. “This format is not supported” gives a different next step than “This file could not be read because its required column is missing.”
Keep these explanations synchronized with actual validation. If the interface claims a format is supported but the parser rejects representative files, the solution is not more reassuring copy. Investigate the accepted behavior and narrow or fix the promise.
Software testing covers deterministic and behavioral checks. Explainability should report their relevant result without claiming that one successful check proves the entire output correct.
Explain change after an incident
When a reported error produces a correction, customers need to know which result changed and why. A general announcement that the model improved may not tell them whether a prior decision requires attention.
Identify the affected scope supported by evidence. Separate confirmed affected reports from potentially affected ones still under review. If the system cannot establish the scope, acknowledge that uncertainty and describe the next investigation step.
Provide a corrected result or a practical remedy where appropriate. Preserve the old version for authorized review according to the retention policy. A customer should not have to remember the previous output to understand the correction.
Use the incident to refine evidence and acceptance cases. If users could not detect a stale source because the date was hidden, expose it and test comprehension. The resulting improvement is a scoped response to a known failure, not proof that all future mistakes are solved.
Preserve meaning when explanations leave the app
A customer may download a report or send a screenshot to a colleague. The explanation should retain the context required to interpret it: source period, account scope, draft status and relevant unresolved conditions. A cropped persuasive recommendation without those limits can create a different claim from the one the interface intended.
Design exports around that handoff. Put critical context near the result rather than exclusively in a remote help panel. A static report should indicate which links require account access and what evidence remains available if the recipient cannot open them.
Avoid embedding live links that reveal more than the intended report or remain usable indefinitely without a stated access policy. The convenience of sharing should be evaluated alongside the data boundary and the customer’s need to recover the evidence later.
Keep explanatory language appropriate to the audience
A technical reviewer may understand a model threshold or an input schema. A merchant may need a plain account of which records were included and what remains to check. The same evidence can support different presentations without changing the underlying claim.
Test terminology with intended users. If they interpret “confidence” as a guarantee or “validated” as professional approval, choose more precise language and explain the scope. Familiar words can carry stronger implications than the developer intends.
Translations and abbreviated mobile views need the same care. Shortening an explanation should preserve the material limit, not only the attractive result. An explanation that loses its caution when space is scarce has changed the customer’s decision evidence.
What Would We Do at Salars?
For a proposed merchant diagnosis app, we would place source date, relevant rows, calculation basis and omitted constraints beside each consequential recommendation. We would identify which statements are observations and which are interpretations.
The merchant could correct a mapping, supply a missing constraint or reject the recommendation. Each action would explain whether it changes the current report, a saved account setting or a reviewed future rule. No automatic global learning would be implied.
A proposed usability review would include a valid result, a stale source, conflicting rows and a missing supplier constraint. Reviewers would ask participants to locate the issue and decide what to do. The review would evaluate correct acceptance and rejection rather than confidence ratings alone; it has not been executed as a Salars product study.
We would keep explanations bound to result versions and preserve an authorized correction history. If the product gained write authority, the explanation and review would reference the exact current action rather than a general narrative about its usefulness.
Explainability earns its place when it helps the customer inspect, challenge and act on the software’s work. The wider AI collection connects that interface to reliable evaluation, permissions and commercial promises.
Sources
- OpenAI: Evaluation best practices — human calibration and model-grader bias; does not validate the proposed interface or recommend dependence on the retiring hosted Evals platform.
Loading comments…