Hallucination

An AI hallucination is a false or unsupported claim that the AI presents as true. It can be a made-up client fact, a changed number, a citation that doesn’t exist, or a claim that an action finished when it didn’t.

The difficult part is that it can sound just like the correct material around it. For client work, I would check the names, numbers, dates, sources, and completion claims against evidence, rather than asking whether the answer sounds confident.

What counts as an AI hallucination?

NIST uses the word “confabulation” for confidently stated false or erroneous content that may mislead someone. The EU-US AI terminology describes it as a false or inaccurate response that can appear plausible.

It doesn’t have to be an entirely invented report. A proposal can name the wrong software version, a report can turn 8.2% into 82%, a summary can cite an article that doesn’t exist, or an agent can say it updated the CRM after the write failed.

But not every bad answer has the same cause. If the model accurately repeats an outdated CRM record, you have a data-quality problem, and a summary that misses the owner’s concern may have an instruction or context problem.

A wrong calculation may also come from a code bug. You need to find which part failed, because “the AI hallucinated” isn’t enough to tell someone what to repair.

Why can AI sound certain when it is wrong?

Language models learn patterns in text and predict what comes next. That training doesn’t attach a true-or-false label to every sentence, so an answer can read smoothly without having factual support.

Bennett proposes one possible explanation in The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest, that models may fabricate because they minimize prediction loss rather than choose the weakest useful claim. He presents it as future work, not a tested explanation of hallucinations.

OpenAI’s September 5, 2025 research also points to how we score models. A test that rewards correct guesses but gives no credit for refusing can favor a model that guesses more, even if that also produces more wrong answers.

In its SimpleQA example, one model was 22% accurate, refused to answer 52% of the time, and was wrong 26% of the time. Another was 24% accurate, refused 1%, and was wrong 75% of the time.

That is a two-percentage-point accuracy gain alongside a 49-point rise in wrong answers in that comparison. So accuracy alone didn’t tell you how often the model would give a wrong answer instead of admitting it didn’t know.

And the wording won’t settle that for the reviewer. A real citation and an invented one can use the same grammar.

Where do hallucinations enter client work?

They can enter wherever an AI answer becomes part of a deliverable or record. Research can invent a source, analysis can explain a pattern the data doesn’t support, and an agent can report success after a timeout or partial write.

Take a monthly client report built from last month’s report, a spreadsheet, and a meeting transcript. The model says retention rose three points after a campaign, which sounds ordinary for that kind of report, but no row in the approved data supports the three points.

The reviewer isn’t careless just because they missed it. It may be one false line inside six correct pages, and you’ve asked them to find it while finishing the rest of their work.

That’s part of the Production Gap, the demo writes a report quickly, while production includes the time spent proving its claims before a client relies on them.

What reduces hallucinations, and what does not?

A clearer prompt narrows the task, but the model can still invent something inside that boundary. A more capable model may make fewer mistakes on your test, though that result still depends on the task, input, and version.

Retrieval from approved sources gives it evidence, but doesn’t prove it read the evidence correctly. NIST’s Generative AI Profile recommends verifying retrieval data, checking sources and citations during testing and monitoring, and reassessing risk after retrieval or fine-tuning changes.

A business rule can block a known invalid value or action, but won’t catch a failure nobody wrote a rule for. An automated checker can flag unsupported or inconsistent claims, but it can miss a false one or reject a true one.

A June 2024 Nature study tested semantic entropy, a signal based on uncertainty in meaning, across 30 task-model combinations. Its mean AUROC was 0.790, compared with three baselines from 0.687 to 0.698.

AUROC measures how well a detector separates cases across thresholds, it isn’t a claim that 79% of your answers will be checked correctly. The result supports using a warning signal to route uncertain answers, not treating everything unflagged as true.

A named person can apply judgment where a mistake costs more, but their review time belongs in the cost of delivery too. None of these controls covers the whole problem alone.

How should a business check AI output?

Pull out the claims that need evidence, names, numbers, dates, prices, citations, quoted terms, client facts, and statements that an action finished. Put each beside the approved source that supports it.

Then match the check to the consequence. You can discard a bad brainstorm idea, but a proposal fee needs checking against the pricing table and a client-specific claim needs the client record.

A legal or financial statement needs a qualified person and a primary source. And if the agent says it sent an email, check the delivery record, not another sentence from the model saying it succeeded.

A March 2026 ACL paper reported a 35.5% increase in factual content after fact-level correction, compared with 10.6% for its sentence-level baseline. That’s a result from that study, not a service-work benchmark, but it supports checking individual claims rather than treating a whole paragraph as one fact.

Count valid refusals separately from errors. “Source not found,” “two records conflict,” and “approval required” can be the right answer for the workflow, so don’t force the model to fill missing evidence with a guess.

Can AI check its own hallucinations?

AI can help flag possible hallucinations, but the final decision still needs evidence outside the generated answer. One model can extract claims and another can compare them with approved records.

Repeating a question can also expose unstable answers. The EMNLP 2023 SelfCheckGPT paper uses disagreement across sampled answers as a warning of possible factual hallucination, but several answers agreeing doesn’t prove they match the outside world.

An eval measures how often the system passes representative cases, a guardrail blocks a known bad value or action, and retrieval supplies evidence. A human-in-the-loop decision handles claims whose cost or uncertainty exceeds what those checks can handle.

The second model still needs the source record, a narrow grading rule, and permission to fail the output. Keep examples of what it misses, because moving the final word to another model doesn’t remove the need to test the checker.

What should happen after a hallucination is found?

Record the false claim, what disproved it, the input and sources, model version, missed check, and where the output went. Then change the source, rule, prompt, eval case, or approval route that let it through.

Operations can mark the bad artifact and send that example to the implementation owner, who updates the check and reruns representative cases. The next reviewer shouldn’t have to discover the same failure in another long document.

In a franchise network’s delivery data, one week with more than 300 generated artifacts showed roughly 95% fewer artifacts marked as containing incorrect information or data after tuning than the same system as first implemented. That was a result from several layers working together, online fact checks, checks against client data, recurring-error tests, and language and logic guardrails.

It wasn’t one model’s accuracy score, and it doesn’t establish a 95% reduction in all errors. It describes flagged artifacts in that comparison, not a result another firm should expect.

Where does hallucination sit on the Delivery Model Ladder?

Hallucination control belongs mainly at Stage 2, Augmented, on the Delivery Model Ladder, where sources are named, routine claims are checked, exceptions go to a person, and failures become test cases.

At Stage 1, a person starts the task and rereads the answer, which can work at low volume. But review cost grows with output, and Stage 3 needs evidence records, automated checks, guardrails, and exception routing as part of delivery.

That assumes an approved source of truth and someone who owns the workflow. Without both, AI can help draft factual work, but keep the final deliverable with a qualified person when a wrong answer cannot be checked or undone.