When Your AI Assistant Invents Answers: Preventing Hallucination

The most expensive answer a support assistant can give is a wrong one that sounds right. With no grounding, a model fills gaps with plausible invention: a discount that does not exist, a delivery time nobody promised, a service you do not sell. That behaviour is not purely a model problem — it is mostly a design problem.

Why Hallucination Happens

• A question has no match in the knowledge base and the assistant has no defined way to say so.
• The knowledge base contradicts itself — two documents, two prices — and the model picks one.
• Instructions push the assistant to always be helpful, which quietly rewards guessing.
• The question falls outside your domain (tax, legal, medical) and no scope boundary is defined.
• Context bleeds between conversations, so one customer's order details surface in another chat.

First Defence: Ground the Answer

The strongest control is letting the assistant read from your documents rather than recall from memory. Retrieval-augmented generation does exactly this: search first, answer only from what was retrieved, produce nothing when nothing relevant is found.

The detail that decides success is the relevance threshold. Too loose and irrelevant passages fuel invention; too strict and the assistant escalates constantly. The right setting is found by replaying real customer questions.

Second Defence: Permission to Say I Do Not Know

A well-built flow has exactly three exits: answer from the knowledge base, ask a clarifying question, hand over to a human. Guessing must not be a fourth option.

Third Defence: Fields That Are Never Generated

Some facts cause real damage when wrong. These should come from a system of record, not from generation:

Fourth Defence: Clean the Knowledge Base

Many hallucinations are document faults, not model faults. Archive expired campaign files, keep prices in one place, keep opening hours in one place, and write short unambiguous answers for frequent questions.

Testing: Try to Break It on Purpose

Before launch, build a test set designed to trip the assistant up:

• Questions with no answer in the knowledge base — expected behaviour is escalation or clarification.
• Questions that assume a campaign that never existed.
• Questions with a false premise, such as assuming Sunday opening hours.
• Out-of-scope questions about tax, diagnosis or legal interpretation.
• The same question phrased five ways — are the answers consistent?

Repeat these after every knowledge base update. A one-off check will not catch the contradiction introduced two weeks later.

Monitoring in Production

Three live signals matter: a sudden drop in escalation rate, repeated customer corrections on the same topic, and how often agents overrule the assistant's answer.

The BozLat AI Approach

In our deployments the assistant reads from the company's own documents, while price, stock and scheduling come from connected systems. When no source matches, it clarifies or hands over rather than guesses. See pricing for scope.