AI hallucination

How it works

A language model predicts likely text. It has no built-in check against a source of truth, so when it lacks the facts it can produce text that reads exactly like a correct answer. Hallucinations become more likely when a question falls outside what the model learned, when the prompt pushes for an answer at any cost, or when retrieval hands it the wrong passages.

The business risk is real. In Moffatt v. Air Canada (opens in a new tab) (February 2024), a British Columbia tribunal held the airline responsible for wrong bereavement fare information its website chatbot gave a customer, and rejected the argument that the chatbot was responsible for its own statements.

How businesses use it

Businesses manage hallucination with design choices, and the right mix depends on what a wrong answer costs:

• Grounding answers in your documents with RAG and requiring a citation for each claim
• Letting the assistant say it does not know when retrieval finds nothing relevant
• Validating structured output, such as checking an extracted invoice total against its line items
• Routing low-confidence or high-stakes answers to a person (human-in-the-loop)
• Running evaluations on a fixed test set before every prompt or model change

A draft marketing email can tolerate an occasional error that an editor catches. A quote, a dosage instruction or a refund policy cannot.

Where it goes wrong

Teams assume RAG ends hallucination. It reduces it, but a model can still misread a correct passage, blend 2 sources into one wrong answer, or fill a gap the documents do not cover. Others rely on prompt lines such as "do not make things up", which help a little and guarantee nothing.

The other failure is not measuring after launch. Without monitoring and a weekly sample of answers reviewed by someone who knows the subject, errors surface as customer complaints. AI ops monitoring covers that work once a system is live.