Why Legal AI Keeps Fabricating Citations

I spent four years as a patent agent before I built agentic systems for a living, so I have a specific allergy: tools that are confidently wrong. In the US alone, there are over 1100 documented legal decisions where a court or tribunal explicitly found or implied that a party relied on hallucinated content¹. That’s not counting all the additional cases where the AI was confidently wrong enough to slip past the judge. In patent prosecution, “confidently wrong” is a missed deadline or a malpractice exposure. The same failure mode shows up in every feature that blindly trusts an AI — it just costs a different amount depending on who’s relying on the output.

Here’s where it stops being someone else’s problem: Every AI feature demos well. That’s the problem. A demo is a controlled environment — your inputs, your happy path, your five-minute window. Production is none of those things. Production has users who don’t know and don’t care about the model’s limits. Production has sources that haven’t been vetted or verified. Production has an infinite timeline for avoiding hallucinations, not a tidy timed demo. In production, your AI feature fabricates its citations, and no one is the wiser until it gets in front of opposing counsel, the judge, or the client.

The real question: What is the cost of a mistake? If it’s marketing copy that’s overhyped, that’s just expected. If it’s a chatbot that points you to the wrong menu, that’s annoying and wastes a minute of the user’s time. You can live with those things. But when the cost is a fine, a ruling against you, a blown investigation, or the loss of a client — those are the mistakes that you can’t tolerate. If that’s not your risk profile, you don’t need me. Save your money.

I take a three-tiered approach to make sure the model never has the authority to write a fact into the record on its own. State doesn’t live in the model’s context, where it can be fabricated. It’s not owned by the model at all. State lives in a backend database, separate from the model’s context and independent of its hallucinations. Tool-use boundaries force the model to follow a particular workflow and respond with structured data, not free text. Validation rejects nonconforming output, enforcing a consistent data model and preventing nonconforming data from reaching the database.

As proof, I built a game moderation system that will play tabletop roleplaying games (TTRPGs) with you. It’s called Zoltar. It’s a low-risk domain, but the engineering principles that keep Zoltar honest apply to high-risk domains. During a playtest, the model made a simple math error. One character, who had 9 hit points (HP), took 5 points of damage. The character should have been at 4 HP and still alive, but the model tried to drop the character to 0 HP and mark it as dead. By comparing the model’s math against the authoritative database, the validation caught the math error and sent it back for a correction.

The boundary that stops Zoltar from fudging math or inventing a die roll is the same one that stops a legal tool from surfacing a citation that doesn’t resolve to a real record. Hit points or case numbers, same boundary — same engineering discipline.

Alternative solutions are available, but they don’t get at the core problems. It’s worth explaining why.

More prompt engineering. Refining the prompt doesn’t address the causes of hallucinations. Hallucinations are the result of the model picking words based on probability distributions without concern for what’s factually true. Prompting shapes tone and format. It doesn’t change the model’s nature or constrain its output against reality. Bloating context to cover edge cases doesn’t increase reliability — it just increases the number of failure modes.

Use a better model. All models hallucinate. You can’t prevent that. You can only catch nonconforming output and filter it out before it reaches your production data and documents. Using a newer model just gives you a false sense of security.

Add a citation-checker. This is better than nothing, and there’s a whole market of these now. But an end-stage checker only finds mistakes that someone thinks to look for. It’s much better to prevent mistakes from reaching production data than to try to weed them out after the fact.

If you’re shipping an AI legal tool or building one inside your own walls, you can’t afford to jeopardize customers’ reputations (or your own) over a hallucination. You can’t risk a fine or an adverse ruling based on fabricated facts.

If that’s your problem, tell me about it.

Footnotes

1. Charlotin, Damien. “AI Hallucination Cases Database.” https://www.damiencharlotin.com/hallucinations/?q=&sort_by=-date&states=USA&period_idx=0. 11 June 2026. ↩