The hallucination trap

6 min read · from the paper

The confidence of a real citation

A legal answer can look reassuring before anyone has checked its reasoning. It names a case, quotes a passage and offers a clear conclusion. Open the citation and the judgment is real. One important question is settled: the source exists. Several others remain. Does the passage support the claim? Does the authority apply to these facts? Has the research missed something that changes the answer?

In The hallucination trap, Samantha Swift, who leads product engineering at Zeno, examines that gap. Her starting point is a personal experience: buying a home in the Netherlands and trying to understand conflicting information about its foundations. As an Australian navigating an unfamiliar legal system, she found sources useful for preparing better questions for her lawyer. Finding evidence helped her participate in the discussion. It did not remove the need for legal advice.

That distinction shapes this first Engineering Note. A reliable research process needs more than a citation that opens. It needs ways to inspect the evidence, expose missing information and revisit the answer.

Research starts before the answer is written

It helps to picture a research folder. Before drafting an answer, a legal AI system must find the documents that could matter, select material from them and bring that evidence into the task. Choices at each stage influence the eventual conclusion.

The paper describes two routes into that folder: looking up a known reference and searching for sources relevant to a problem. Search returns candidates. For longer sources, a model may select original passages to pass to the answering model. The connection to the original document is retained, allowing the lawyer to inspect what was selected and the context around it.

Using original passages matters because another paraphrase would introduce another opportunity to change the meaning. But selection itself can go wrong. A relevant qualification might sit outside the selected extract. Giving the model every full judgment is not a universal solution either: there is a trade-off between the depth of each document and the breadth of material that fits into the available context.

The wording of the question also matters. A follow-up can send the research down another path and bring different evidence into view. Research is therefore an iterative process, not a single search followed by a finished answer.

Following the connections between sources

A case rarely stands alone. It may cite an earlier authority, discuss a related ruling or depend on legislation that has since changed. The paper explains how Zeno's knowledge graph helps the research follow recorded connections between legal sources.

Those connections can surface material that a first search did not find. They do not settle what that material means. A court may follow an earlier judgment, distinguish it or refer to it for a limited purpose. The relationship points to something worth reading; the legal significance still depends on the source and the question.

This is why the research process returns to original material. A source being connected, retrieved or cited is a reason to inspect it, not proof that a conclusion is correct.

A failed lookup is a clue to investigate

One engineering example in the paper begins with a log of unsuccessful legislation lookups. An empty result might seem to indicate missing coverage. Investigation showed that this was not always the explanation: sometimes the legislation was already present, but the requested reference did not resolve.

A separate check for certain legislation lookups can establish that a law title is recognised even when the requested article is not found. That narrows the investigation. The cause could involve an article number, a naming variation, a date, retrieval behaviour or a gap in the available material.

The important step is to distinguish those possibilities. Adding more documents will not fix every reference error. Changing a prompt cannot retrieve a document that is absent. The paper describes a loop of recording failures, investigating their causes, making targeted changes and testing again, including checks for regressions.

It also makes a measurement distinction that matters beyond this example: the rate of one particular lookup failure is not an overall hallucination rate or an overall measure of legal accuracy.

The source can be right while the answer is wrong

The paper separates several failure modes that can otherwise disappear under the single word “hallucination”:

• An invented source: the reference cannot be verified.
• A misrepresented source: the document exists, but the answer changes or overstates its meaning.
• The wrong context: an authority is used without accounting for relevant jurisdiction, timing or factual differences.
• Assumed facts: the answer treats something unknown or disputed as established.
• An unsupported conclusion: the reasoning goes further than the cited material supports.
• Missing material: an exception, qualification or contrary authority never enters the analysis.

These problems require different checks. Verifying that a document exists is a different task from assessing whether it supports a proposition. Assessing support is different again from deciding whether that proposition resolves the client's matter. Models used to select evidence or assess support can also make mistakes.

Make the unknowns visible

Even strong sources cannot repair an answer built on an invented fact. If the analysis assumes that a buyer received no warning, but the record does not establish that, the conclusion may rest on a premise that has never been tested.

The paper recommends asking the system to identify facts that could change its conclusion. One practical instruction is:

If a missing fact could change your conclusion, ask me about it or explain how the answer depends on it. Don't assume it.

This gives the lawyer something concrete to resolve: an unanswered factual question, a disputed assumption or a conditional conclusion. It also gives the next research step a clearer direction.

What to check before using the work

The practical outcome is a review process with several questions, rather than a single green light beside a citation. Separate the supplied facts from assumptions. Open the original sources and read the relevant context. Check whether the authorities fit the jurisdiction, date and facts. Look for qualifications, contrary material and gaps. Resolve what could change the answer before relying on it.

Zeno's engineering work aims to make that process easier to inspect and improve. The paper is equally clear about the remaining responsibility: a supported passage or a real citation does not replace a lawyer's review of the reasoning and its application to the matter.

About the author

Samantha Swift leads product engineering at Zeno, working on agentic workflows and scalable machine-learning systems. This article draws on her September 2026 Engineering Note, The hallucination trap. Download the full 11-page paper for the examples, process diagrams and references behind the discussion.