What is AI hallucination?
An AI hallucination is an output from a large language model (LLM) that sounds right but is false, unsupported or invented: a delivery date that was never promised, a discount that does not exist, a refund policy the business never had, a link to a page that is not there. The term covers both factual errors and fabricated details, and its defining feature is confidence — the wrong answer reads exactly like a right one.
It happens because an LLM generates text by predicting likely next tokens from patterns in its training data and whatever is in its context window. It has no built-in sense of which statements are verified. When the prompt lacks the fact it needs — today's price, this customer's order, a policy changed last week — the model tends to produce the most plausible completion rather than saying it does not know. Long chats, ambiguous questions, leading questions from the user and instructions such as "always give an answer" make it more likely.
Hallucination is the failure mode; the controls are separate. Retrieval-augmented generation (RAG) grounds answers in your knowledge base, tool calls fetch live data such as order status, AI guardrails block or rewrite unsupported claims, and LLM evaluation measures how often it still happens. No current technique eliminates it, so customer-facing bots also need a clear route to a person for anything that commits the business.
Key takeaways
- An AI hallucination is a fluent, confident answer that is false or not supported by any source.
- Language models predict likely words, not verified facts, so gaps in their context get filled with plausible guesses.
- Grounding answers in your own documents (RAG), permitting "I don't know" and citing sources all reduce hallucination.
- In 2024 a Canadian tribunal held an airline liable for a refund policy its website chatbot misstated.
- No technique removes hallucination entirely: measure it with evaluations and keep a path to a human.
Figure 1
Same question, two answers: spot the hallucination
Compared on
Hallucinated answer
Fluent, confident, unsupported
Grounded answer
Built from your own sources
- Hallucinated answerPatterns in training dataGrounded answerA retrieved passage or a live tool result
- Hallucinated answerGuesses the most plausible answerGrounded answerSays it doesn't know and hands over
- Hallucinated answer"Returns within 30 days" — typical, not your policyGrounded answer"7 days for unworn items [policy-4]" — your policy
- Hallucinated answerInvented or out of dateGrounded answerFetched from today's catalogue or offers sheet
- Hallucinated answerNone, or a made-up linkGrounded answerA passage id or document link anyone can check
- Hallucinated answerThe customer, after the factGrounded answerGuardrails and evaluations, before sending
- Hallucinated answerComplaints, refunds, legal liabilityGrounded answerLimited to what your sources say
Verdict
The model is the same on both sides. What changes is the architecture: grounding, permission to say "I don't know", checks before sending and a path to a person.
Verdict: The model is the same on both sides. What changes is the architecture: grounding, permission to say "I don't know", checks before sending and a path to a person.
Example: An offer that never existed
During Dhanteras week, a customer asks a Jaipur jewellery store's WhatsApp bot: "Is making charge waived on gold coins today?" With no current offer in its context, the bot replies that making charges are waived on all coins — a promotion the store never ran. After the complaint, the team rebuilds it: offers now come from a sheet updated every morning and retrieved for each question, the system prompt forbids stating prices or discounts not found in retrieved text, and a check compares every number in a draft reply with the retrieved passages. Asked again, the bot answers that today's offer covers making charges on jewellery above 10 grams, that coins are not included, and offers to share coin prices — quoting the sheet — and hands anything the sheet does not cover to staff.
Why it matters
Why does AI hallucination matter?
On WhatsApp, a bot's answer arrives in the same thread, under the same business name and badge, as a message from your staff — customers read it as your word. A hallucinated price, warranty term or refund rule becomes a complaint, a chargeback or a legal claim, and a screenshot of it can travel far. Reducing hallucination is less about which model you pick than about architecture: what the bot may answer, from which sources, how answers are checked and when it must hand over.
How does AI hallucination work?
The model lacks the fact
The answer depends on data that is not in the prompt: live stock, the customer's own order, an offer that started this morning, a policy changed after the model was trained.
It fills the gap fluently
Trained to produce helpful, complete text, the model writes the most probable answer — a typical return window, a likely price — with nothing in the output that signals it guessed.
Ground it in your sources
Retrieve the relevant knowledge-base passages, call tools for live data, and instruct the model to answer only from that context and to name the passage it used.
Make "I don't know" an accepted answer
Give the model an explicit way out — "I'm not sure, let me connect you to our team" — and route those cases to a human handoff instead of treating them as failures.
Check before sending
Use guardrails to flag prices, numbers, dates and policy claims that do not appear in the retrieved context, and send high-stakes answers to an agent for approval.
Measure all the time
Run an evaluation set of real questions with known answers before every prompt, model or content change, and review a sample of live conversations each week.
Common mistakes with AI hallucination
Letting the bot answer from general knowledge
Asked about your returns policy without the policy in its context, a model writes a typical one. Retrieve your actual policy every time, or have the bot decline.
Forbidding "I don't know"
Prompts such as "always answer the customer" push the model to invent. Make uncertainty an accepted outcome that routes to a person.
A stale knowledge base
RAG grounds answers in whatever you indexed. If last season's offers are still in the index, the bot quotes them faithfully and is still wrong. Expire and re-index content on a schedule.
Testing only the happy path
Hallucinations surface at the edges: questions just outside scope, leading questions ("so delivery is free, right?"), mixed languages. Put those in the evaluation set.
Generated text making commitments
Refunds, exceptions, and legal or medical questions should never rest on generated text alone. Route them to a person with the draft as a suggestion.
AI hallucination in code
// 1. Ground the model: answer only from retrieved passages, allow "I don't know"const SYSTEM = ["You are the WhatsApp assistant for a jewellery store.","Answer ONLY from the passages inside <context>.","If the answer is not there, reply exactly: I'm not sure, let me connect you to our team.","Never state a price, discount, date or policy that is not in <context>.","End with the id of the passage you used, like [offer-12].",].join("\n");// 2. Before sending: every figure in the draft must appear in the retrieved textfunction unsupportedFigures(draft, passages) {const source = passages.map((p) => p.text).join("\n");const claims = draft.replace(/\[[^\]]*\]/g, ""); // ignore citation ids like [offer-12]const figures = claims.match(/\d[\d,.]*/g) ?? [];return figures.filter((n) => !source.includes(n));}export async function answer(question, conversation) {const passages = await retrieve(question, { topK: 4 });const draft = await llm.generate({ system: SYSTEM, context: passages, user: question });const unsupported = unsupportedFigures(draft, passages);if (passages.length === 0 || unsupported.length > 0) {return handoff(conversation, "unsupported figures: " + unsupported.join(", "));}return sendText(conversation.waId, draft);}
A crude but useful guard: no retrieved source, or a number the source doesn't contain, means a person checks first.
Best practices for AI hallucination
- Keep facts in data, not in prompts: prices and stock through tool calls, policies through retrieval.
- Show the source for high-stakes answers so agents and customers can check them.
- Use low-randomness settings for factual replies; they reduce variation, though not the root cause.
- Track a hallucination rate on sampled conversations as a standing quality metric.
AI Hallucination FAQs
- Why do AI chatbots make things up?
- Because language models generate the most likely text, not verified facts. When the information they need is missing from their context, they tend to fill the gap with a plausible guess and present it as confidently as a real fact.
- Can RAG eliminate hallucinations?
- No, but it reduces them when retrieval finds the right passages. The model can still misread or over-generalise a passage, or answer when retrieval found nothing, so pair RAG with "I don't know" rules, guardrails and regular evaluation.
- Is a business responsible if its chatbot gives wrong information?
- It can be. In Moffatt v. Air Canada (2024), a Canadian tribunal held the airline responsible for a bereavement-fare refund policy its website chatbot misstated, rejecting the argument that the bot was answerable for itself. Treat your bot's answers as your own statements.
- What is the difference between hallucination and guardrails?
- Hallucination is the problem; guardrails are one of the controls. Guardrails filter inputs and outputs — blocking unsupported claims, off-topic requests or personal data — while grounding tackles hallucination at its source and evaluation measures how often it still happens.