A support engineer files a ticket: the assistant made something up. It goes to the team as a hallucination, which points everyone at the model, which leads to prompt changes, a temperature adjustment, and eventually a proposal to try a bigger model.
In a retrieval system, most of those tickets are not generation failures. The model answered from nothing because nothing useful was put in front of it, which is a different bug in a different component with a different fix. The reason it keeps getting misdiagnosed is that all three possible failures produce the same symptom, and almost nobody logs enough to tell them apart.
Three questions, not one
Every answer in a retrieval system passes through three independent gates, and each can fail on its own:
- Is the answer in the corpus at all? If it is not, no amount of tuning helps, and the correct output is a refusal rather than an answer.
- Did retrieval put it in the context? The answer exists, the search did not surface it in the top results.
- Did the model use it? The right passage was in the context and the model answered from its own weights anyway, or contradicted the passage.
Those are three different engineering problems: a content problem, a search problem, and a prompting problem. Rolling them into “hallucination” guarantees you work on the wrong one, because the third is the only one prompt changes affect and it is usually the rarest.
| What you observe | Which stage | What actually fixes it |
|---|---|---|
| Confident answer, no source in the corpus contains it | Corpus | Add the content, or make refusal the required behaviour when retrieval returns nothing above threshold |
| The right document exists, was not retrieved | Retrieval | Chunking, hybrid search, reranking. Not the prompt. |
| Right passage in context, answer contradicts it | Use | Prompt structure, citation requirement, smaller context |
| Answer is right but cites the wrong chunk | Use | Force citation from the provided ids and validate them programmatically |
| Was correct last month, wrong now, nothing changed in the prompt | Retrieval | Index or embedding drift. Check the model id stored with the vectors. |
Chunking destroys answers before search runs
Splitting documents at a fixed character count is the default in every tutorial and it is a lossy transformation applied before anything else happens. A boundary through the middle of a table separates the row from its header. A boundary after a definition’s first sentence leaves the qualifying clause in a different chunk. Neither chunk is retrievable as an answer to the question the document actually answers.
What helps, roughly in order of payoff:
- Split on structure, not length. Headings, list boundaries, table boundaries, code fences. Length becomes a cap rather than the rule.
- Keep the parent reachable. Retrieve the chunk, then give the model the surrounding section. The small unit is good for matching, the large unit is good for answering, and there is no reason they have to be the same object.
- Prepend the context the chunk lost. Document title and heading path at the top of each chunk. Cheap, and it fixes a large class of near-misses where the chunk text alone is ambiguous.
- Never split a table. If it does not fit, keep it whole and accept the outlier.
Pure vector search cannot find an identifier
This one causes more confusion than anything else, because the failure looks stupid. A user asks about error E4021, the corpus contains a page about error E4021, and retrieval returns three pages about other errors.
Embeddings measure semantic similarity, and an opaque identifier carries almost no semantic signal. E4021 and E4022 sit at nearly the same point in the space; so does “error code” in general. The model is behaving exactly as designed. It is the wrong tool for exact-match retrieval, and the same applies to SKUs, version numbers, function names, customer identifiers, and legal clause references.
Hybrid retrieval is not an optimisation here, it is a correctness requirement: run a lexical search alongside the vector search, merge the candidate sets, then rerank the union with a cross-encoder that reads the query and the passage together. The lexical half finds the identifier, the vector half finds the paraphrase, and the reranker decides which of the two mattered for this query.
If a user can paste a string from your product into the box, you need lexical search. They will.
Re-embedding is a migration, not a refresh
Vectors from two different embedding models are not comparable. Distances between them are arithmetic that runs without error and means nothing. So the moment you change embedding model, or the provider silently updates one behind the same name, an index containing both is quietly broken: no exception, no failed job, just retrieval quality falling off in a way that shows up weeks later as “the assistant got worse”.
The discipline is small and worth enforcing:
# Store the model identity with every vector, and refuse to mix.
# Re-embedding writes a new index and swaps an alias, so rollback is free.
record = {
"chunk_id": chunk_id,
"doc_id": doc_id,
"heading_path": ["Billing", "Refunds", "Partial refunds"],
"text": text,
"embedding": vector,
"embedding_model": "text-embed-3-large", # not optional
"embedding_version": 4,
"content_hash": sha256(text), # skip unchanged chunks
"indexed_at": now(),
}
def search(query, index):
if index.embedding_model != CURRENT_MODEL:
raise ConfigError(
f"index built with {index.embedding_model}, "
f"querying with {CURRENT_MODEL}"
)
...That single exception converts a silent quality regression into a startup failure, which is the trade you want. The content hash earns its place separately: it makes re-indexing incremental, so a corpus refresh stops being an all-day job that people avoid running.
Measure retrieval without the model in the loop
The useful property of the three-stage decomposition is that stage two is measurable on its own, cheaply, deterministically, and without a language model anywhere in the test. Take a set of real questions, label which chunks genuinely contain the answer, and measure how often retrieval returns at least one of them in the top k.
# A few hundred labelled questions is enough to make chunking and
# search changes decidable instead of a matter of opinion.
def recall_at_k(cases, retrieve, k):
hits = 0
for case in cases: # case.relevant_chunks: set of ids
got = {c.id for c in retrieve(case.question, k=k)}
if got & case.relevant_chunks:
hits += 1
else:
log.info("miss", q=case.question, expected=case.relevant_chunks,
got=list(got)) # the misses are the work queue
return hits / len(cases)Run it on every change to chunking, embedding model, k, or the retrieval mix. It is the same argument as treating evals as a test suite rather than a leaderboard: a number that moves when you make a change, on cases you chose because they matter, beats a benchmark that moves for reasons you cannot attribute.
The production counterpart is one log line per answer: the query, the retrieved chunk ids with scores, and which ids the model actually cited. With that, a ticket saying “it made something up” becomes a lookup that tells you which of the three stages failed, and the argument about whether to change the prompt ends in about a minute.