This chapter adapts Part VI's validation discipline - frozen baseline, held-out packs, conjunctive gates - to LLM-specific failure modes: ticker hallucination, JSON validity, Brier on stated probabilities, and hash-locked holdout packs. The framework draws directly on the validation chapters of Part VI - walk-forward, frozen-baseline, conjunctive gates - but adapts them to the specific failure modes of language models.
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
The frozen baseline: why one model must never be promoted
The most counterintuitive discipline in LLM evaluation: one specific model version must be designated as the eternal baseline and explicitly forbidden from being promoted to production. Every candidate is scored against this frozen baseline; promotion requires beating it on every category the baseline was scored on. The argument: a baseline that changes over time is no baseline. If the comparator is "yesterday's production model," then a long sequence of small regressions can ratchet quality downward - each step looks tied or marginally better than its immediate predecessor, but the cumulative effect after twenty cycles is a model meaningfully worse than the version of two years ago. A frozen baseline anchors the comparison: the promotion question becomes "does this candidate beat the version we chose, with care, as our quality anchor?" - not "does this candidate beat the previous candidate?"
Held-out sets: the discipline of never touching the eval pack with training
The second discipline is even more important: the prompts used to evaluate the model must never have been seen by the model during training. This is the held-out property - the same one that Part VI demanded for backtest data. The implementation is unglamorous. The eval pack lives in a separate directory tree from the training corpus. A small check, run before every training cycle, asserts that no file appears in both. The check is conservative: it hashes filenames, prompts, and even short n-grams to detect partial overlap from copy-pasted examples. A held-out pack of fewer than fifty items is statistically thin - the verdict on a candidate can flip from promote to reject based on a few prompts that happened to fall on one side of the threshold. A pack of two hundred to a thousand items is more honest.
Brier calibration of probability-of-event answers
Part VI introduced the Brier score as the right measure of probability calibration for trading systems. The same measure applies to language models that produce probability-of-event answers. A prompt like "what is the probability that this company beats earnings estimates next quarter?" invites the model to produce a number. The Brier score on a held-out set of such prompts - with each prompt paired against the realised outcome - tells you whether the model's stated probabilities are calibrated. A model that says "60%" for events that happen 60% of the time has a good Brier score; a model that says "90%" for events that happen 60% of the time has a worse Brier score even if its directional accuracy is the same. The Brier check is the honesty test for language models in finance - directional accuracy is easy to optimise, but calibrated confidence is what actually protects position sizing.
Conjunctive vs additive gates
The promotion gate must be conjunctive, not additive. A candidate that improves on five categories but degrades on one is rejected; the gate is and, not average. The temptation to use an additive gate ("compute a weighted average of all categories; promote if positive") is real. The argument feels statistical - "we should let small wins offset small losses." It is wrong for a simple reason: the worst category is what the user experiences. A model that hallucinates 5% of tickers (versus 2% baseline) ships hallucinations to users - the fact that its JSON validity improved by 1% does not protect them. The conjunctive gate enforces "do no harm on any axis": improvements anywhere are welcome, but regressions anywhere block promotion.
Definition
A frozen baseline is a specific model version, deliberately preserved on disk under an unchanging name, against which every candidate is evaluated. The baseline is never promoted to production; it exists only as a stable comparator. Promotion of a candidate requires the candidate to match or beat the baseline on every category measured. Without the frozen baseline, the promotion process measures candidates against each other and inherits all the drift that creates.
Worked picture
A candidate model scored: - Ticker grounding: 99.2% (baseline 98.7%, +0.5 - pass) - JSON validity: 99.5% (baseline 99.3%, +0.2 - pass) - Sentiment direction: 87.4% (baseline 84.1%, +3.3 - pass) - Brier calibration: 0.241 (baseline 0.219, +0.022 - fail) - Latency p95: 2.1 s (baseline 2.0 s, 1.05× - pass) The candidate beats baseline on four of five categories and looks superficially better. The gate refuses promotion because of Brier. Investigation reveals the new alignment training pushed the model toward more confident probability statements, which sounded better in informal review but priced too tight on uncertain events. The corpus is adjusted to include more uncertain-outcome examples; a follow-up candidate trained on the revised corpus passes all five categories. The Brier check saved the platform from shipping a model whose confidence statements would have produced systematically oversized positions.
Figure. Candidate vs. baseline Brier calibration score — the one category among five that fails, blocking promotion despite four passes.
What breaks in production
A common architecture mistake: build a "model quality score" that weights the seven category scores into a single number, and promote whenever the composite improves. This is the additive gate dressed up in respectable clothing. The composite-score architecture is wrong for the same reason the additive gate is wrong: it lets a marginal win on a low-weight category buy permission to ship a regression on a high-weight one. Users do not experience the composite; they experience the regression. The discipline: keep the categories disaggregated, surface them all, and require every one of them to clear. A candidate that lacks the discipline to improve everywhere is a candidate the system is not ready to ship.
Key idea
Evaluation discipline is what separates a language model that someone trained from a language model that is fit for production. A frozen baseline that never moves. A held-out pack big enough for the verdicts to mean something. A Brier check that catches confidence drift. A conjunctive gate that refuses to ship regressions anywhere. A cryptographic lock that makes every verdict reproducible by hash. A shadow protocol that tests routing changes against live traffic before they affect users. Most of the cost of running a serious LLM system is paid here, not in training; the few engineers who internalise this end up being the ones whose models actually compound.