This chapter closes Arc II with the verify pass: a deterministic post-hoc check that compares extracted numbers and tickers in the answer against the tool haystack - microseconds, no second LLM, loggable in production. Hallucinations come in two flavours that respond to different defences.
Figure 1. Concept map for this appendix entry: fight, desk translation, what to lock.
The two kinds of hallucination
Hallucinations come in two flavours that respond to different defences. Catastrophic hallucination: the model invents an entity that does not exist - a ticker symbol, a company, a price that has no source in the data. The user has no way to verify; if they trust the answer, they will act on fiction. Drift hallucination: the model takes a number that was in the tool results and modifies it slightly (rounds it, adds a decimal place, multiplies by a factor) when narrating the answer, presenting the modified number as if it were the original. The user sees a plausible-looking number that does not actually match the data.
Numeric overlap as a deterministic verifier
The simplest verify pass that works is also the cheapest: 1. Extract all numeric tokens from the model's answer with a regex. 2. Extract all ticker-shaped tokens (1-7 uppercase letters with an optional exchange suffix), minus a stop-list of common acronyms (USD, RSI, EBITDA - things that look like tickers but aren't). 3. Flatten every tool result to a JSON string - the haystack. 4. For each extracted item, check whether it appears in the haystack, allowing for normalisation - 142.30 matches 142.3, 1,234,567 matches 1234567, 1.4% matches 0.014.
LLM-on-LLM verification and why it shifts the problem
For completeness, consider the LLM-as-verifier design. The verifier model reads the answer alongside the tool results and returns a verdict plus a list of specific issues. The architecture is more powerful - the verifier can detect semantic issues a regex misses ("the answer claims earnings beat estimates, but the tool returned a miss") - but pays three costs: 1. Doubled inference. Every answered request requires a second LLM call. At scale this doubles the GPU bill. 2. Verifier hallucinations. The verifier produces its own false positives and false negatives. Without an external check on the verifier, you have moved the hallucination problem one model deeper. 3. Latency. The second call adds hundreds of milliseconds to the user-visible latency. A hybrid design - deterministic numeric check always, LLM verifier only when the deterministic check passes and the answer has high-stakes content such as a recommendation or a P&L claim - captures most of the value without paying the full cost.
Confidence downgrades, not bounces
A verifier that rejects the answer - refusing to ship it to the user - has obvious appeal but creates a worse failure mode: the user sees nothing. "No answer because verification failed" is, from the user's perspective, indistinguishable from "the system is broken." The better design is to ship the answer with a downgraded confidence and a UI affordance that says "some claims in this answer could not be verified against the source data." The user gets the substance, learns to read with appropriate skepticism, and can re-ask if they want a stricter pass. The discipline: if the verify pass passes, confidence is high with no UI warning; if it fails on a small fraction of items, confidence drops to medium with a soft warning; if it fails on most items, confidence drops to low with a prominent warning and a suggestion to re-ask.
Definition
A verify pass is a deterministic, post-hoc check that the entities and numbers in the model's final answer are supported by the tool results the model consulted. Supported is defined operationally: the number appears in the JSON-serialised tool output, possibly after normalisation (commas stripped, trailing zeros dropped, percentages converted). The verify pass returns a boolean (supported / not supported) and a list of unsupported items.
Worked picture
A user asks "summarise today's screen for me." The model calls a screen-results tool, receives a list of 25 names with metrics, and produces a three-paragraph summary mentioning seven of them by ticker, with metrics. The verify pass extracts seven tickers and twelve numeric values from the summary. All seven tickers are present in the tool output. Eleven of the twelve numbers are present; the twelfth is a number the model rounded ("2.4%" in the summary; "2.43%" in the data). The overlap is 18 / 19 = 0.95 - well above the 0.6 threshold. Confidence stays high. No warning. Now imagine the model also included a sentence saying "the average move was 1.8%" - a number the model computed from the data but that does not appear literally in the haystack. The overlap drops to 18 / 20 = 0.90. Still above threshold; confidence stays high. But the engineer reviewing the logs notices the unsupported number and decides to add an "average move" derived field to the tool output, so future calls find it in the haystack directly. The verifier hasn't blocked anything - it has identified a small data-shape improvement that compounds over future calls. This is the verify pass operating as designed: catching the easy cases automatically and surfacing the hard cases for human attention.
What breaks in production
A subtle danger: a strong verifier can make a weaker generator look acceptable. "The model is fine; the verifier catches its mistakes." This is true until the verifier is wrong, at which point you have two failure modes stacked. The discipline: the verify pass is a safety net, not a quality replacement. Improve the underlying model; treat the verifier as a final check; and never let the verifier become a justification for accepting a generator that you know is producing hallucinations at a meaningful rate.
Key idea
The verify pass is the discipline that converts a model that uses tools into a model whose factual claims are grounded in tool data. Use the cheapest verifier that works (regex numeric overlap), downgrade confidence rather than reject, and treat any persistent verify-failure pattern as a data-shape problem to fix at the source, not a model defect to live with.