Hallucination

An AI hallucination is a false or unsupported claim that an AI presents as true. It may invent a client fact, change a number, cite a source that does not exist, or say an action finished when it did not. In OpenAI’s September 2025 SimpleQA example, one model was 22% accurate, abstained 52% of the time, and was wrong 26% of the time. Another was 24% accurate, abstained 1%, and was wrong 75% of the time. A two-point accuracy gain came with a 49-point rise in wrong answers. For a service firm, the check belongs on the claim, not the tone. Important names, numbers, dates, sources, and completion claims in client work need approved evidence or a named person.

What counts as an AI hallucination?

The claim is a hallucination when it looks plausible but is false or unsupported. NIST uses the more precise word “confabulation” for confidently stated erroneous or false content that may mislead a user. The EU-US AI terminology describes confabulation as a false or inaccurate response that can appear plausible.

The error can be small. A proposal can name the wrong software version. A report can change 8.2% to 82%. A research summary can cite an article that was never published. An agent can say it updated a CRM after a write failed. Each statement fits the shape of normal work, so someone may carry it forward.

Not every bad output is a hallucination. A correct answer from an outdated CRM record is a data-quality failure. A summary that leaves out the owner’s concern may be an instruction or context failure. A wrong calculation may come from a code bug. The repair depends on the failure.

Why can AI sound certain when it is wrong?

Language models learn patterns in text and predict what comes next. Training does not attach a true-or-false label to every sentence in that text. Fluent wording and factual support remain separate properties.

Bennett has proposed one possible explanation. Models may fabricate because they are trained to reduce prediction loss rather than to choose the weakest useful claim. He presents this as a direction for future work in The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest, not as a tested finding.

OpenAI’s September 5, 2025 research points to an evaluation problem too. Many tests reward a correct guess and give no credit for refusing. A model can raise its accuracy by guessing more, while also producing many more confident errors. That is what the 22% and 24% SimpleQA comparison shows.

The tone does not help a busy reviewer. A real citation and an invented citation use the same grammar. Confidence in the wording is not a confidence score.

Where do hallucinations enter client work?

They enter whenever AI output becomes part of a record or deliverable. Drafting can invent a client fact. Research can fabricate a source. Analysis can change a number or explain a pattern the data does not support. An agent can claim a task finished after a timeout or partial write.

Consider a monthly client report. The model has last month’s report, a spreadsheet, and a meeting transcript. It writes that retention rose three points after a campaign. The sentence sounds normal because each part belongs in that kind of report. If no source row supports the three points, the sentence is still a hallucination.

The reviewer is not careless for missing it. The false claim may be one line inside six correct pages. This is part of the Production Gap: a demo produces a clean report quickly, while production spends the recovered time proving claims before they reach a client.

What reduces hallucinations, and what does not?

Each control changes one part of the failure path. None closes the whole path.

Control	What it changes	What remains
Clear prompt	Narrows the requested task and output	The model can still invent a claim inside the boundary
More capable model	May reduce errors on the tested task	Rates still vary by task, input, and model version
Retrieval from approved sources	Gives the model relevant evidence	The answer can misread, combine, or ignore that evidence
Business rule	Blocks a known invalid value or action	Unknown failure patterns pass the rule
Automated checker	Flags claims that look unsupported or inconsistent	The checker can miss a false claim or flag a true one
Named human approval	Applies judgment where a mistake costs more	Review time becomes part of the delivery cost

Retrieval moves an answer closer to evidence. It does not prove the answer is true. NIST’s Generative AI Profile recommends verifying retrieval data, reviewing sources and citations during testing and monitoring, and reassessing risk after retrieval or fine-tuning changes.

Automated detection has the same limit. A June 2024 Nature study tested semantic entropy across 30 task-model combinations. Its mean AUROC was 0.790, compared with three baselines between 0.687 and 0.698. That is a useful signal for routing uncertain answers. It does not prove that the rest are true.

How should a service firm check AI output?

Review the claim, not the paragraph around it. Pull out names, numbers, dates, prices, citations, quoted terms, client facts, and statements that an action completed. Attach each claim to an approved source.

Then route by consequence. A wrong brainstorm idea can be discarded. A proposal fee needs a deterministic check against the pricing table. A client-specific claim needs the client record. A legal or financial statement needs a qualified person and a primary source. A claim that an email was sent needs a delivery record, not another model sentence saying it succeeded.

Fact-level checking is becoming measurable. A March 2026 ACL paper reported a 35.5% increase in factual content after fact-level correction, compared with 10.6% for its sentence-level baseline. That result is not a service-work benchmark. It supports splitting a paragraph into claims before checking what is true.

Count abstentions, or valid refusals, separately from errors. “Source not found,” “two records conflict,” and “approval required” can be correct workflow results. A forced answer turns missing evidence into invented certainty.

Can AI check its own hallucinations?

AI can flag possible hallucinations. The final decision needs evidence outside the generated answer.

One checker can extract claims while another compares them with approved records. Repeating a question can expose unstable answers. The EMNLP 2023 SelfCheckGPT paper found that sampled answers tend to agree when the model has stable knowledge and diverge on invented facts. Disagreement is a useful warning. Agreement does not prove the claim against the outside world.

An eval measures how often the whole system passes representative cases. A guardrail blocks a known bad value or action. Retrieval supplies evidence. A human-in-the-loop decision handles claims whose cost or ambiguity exceeds the automated boundary. Each layer catches a different failure.

A second model grading the first is still a model. Give it the source record, a narrow rubric, and permission to fail the output. Keep examples of misses. If one checker has the last word, the firm has moved the same risk one step down the workflow.

What should happen after a hallucination is found?

Record the false claim, the evidence that disproved it, the input and source set, the model version, the missed check, and where the output went. Then change the source, rule, prompt, eval case, or approval route that let it pass.

Each miss should improve the checking layer. Operations marks the defective artifact. The implementation owner receives the example, updates the check, and sends representative cases through it again. The next reviewer should not have to find the same failure in another long document.

Tuning can take real time. In a franchise network’s delivery data, more than 300 generated artifacts in one week showed roughly 95% fewer artifacts marked as containing incorrect information or data after tuning than the same system as first implemented. The result came from layers: online fact checks, checks against client data, recurring-error tests, and language and logic guardrails. It was not a score for one model or a benchmark another firm should expect.

Where does hallucination sit on the Delivery Model Ladder?

Hallucination control belongs mainly at Stage 2, Augmented, on the Delivery Model Ladder. Approved sources are named, routine claims are tested, exceptions go to a person, and failures become new eval cases.

At Stage 1, Enhanced, a person starts the task and rereads the answer. That can work at low volume. The control lives in individual attention, so review cost grows with output. At Stage 3, AI-native, more work moves without a person touching every item. Evidence records, automated checks, guardrails, and exception routing become part of delivery.

This assumes an approved source of truth and an owner for the workflow. Without both, AI may draft factual work but cannot produce the final deliverable. When a wrong answer cannot be checked or undone, keep the task with a qualified person.