Models propose, and are bounded

Here is a systems problem I keep running into when people talk about letting AI decide after the model is trained.

You can hand an AI the same data, the same training loss, the same holdout checks, and still not get a unique decision ready for capital. The board looks flat. Eighteen out of twenty seeds pass. Someone says we are done. Then two seeds that both won disagree on the number that actually moves money.

Chapter 1 used Bertrand's Paradox to say you cannot ask for an unbiased answer before you lock the ruler. It caged that with frames, constraints, recipes, gap pricing, and multi-agent veto. This post is the twin after learning starts. Even inside one locked frame, training success is not the same thing as a unique answer.

Definitions, because this gets waved away as seed noise

An ML pipeline is the full recipe used to train and check a predictor: filters, features, architecture, random seeds, early stopping, holdout slices. That pipeline is underspecified when many different predictors clear the same checks about equally well. D’Amour et al. made this precise at production scale in “Underspecification Presents Challenges for Credibility in Modern Machine Learning” (arXiv:2011.03395): pipelines that look equivalent on standard holdout can separate hard once you care about robustness, fairness, or a shifted world.

Those predictors form a Rashomon set (Breiman’s name for many good stories of the data; later formalized as predictive multiplicity. Marx, Calmon, and Ustun, and related work on disagreement rates and “tethered” intervals over near-winners): near-optimal models that disagree on the decisions that matter. They only look interchangeable if the only question you ask is average error on yesterday’s quiet distribution. Ask the question that spends, and the set fans out. Recent work (2024–2025) on Rashomon PDPs and set visualization only underscores the desk lesson: treating one champion as “the model” is a reporting choice, not a uniqueness theorem.

That is Bertrand after learning. Bertrand had one question and several valid ways to define random. Underspecification has one training win condition and several near-equal ways to satisfy it. The math can look clean in every member of the set. The decision object is still not unique.

Why train harder often does not close the set

People reach for more epochs, bigger nets, tighter in-distribution accuracy. Those moves help when the failure was under-fitting. They often leave the freedom intact when the failure is multiplicity.

If validation only checks an average that many predictors can match, the optimizer has no reason to collapse onto one decision-relevant answer. Same lesson as maximum entropy without geometric constraints. Optimization fills the freedom it was never told to remove.

There is a second trap: the checks can be easy in the wrong place. Two models can agree on common cases and disagree on the rare structure, the jump week, or the exact contract you mark. Leaderboard culture rewards the easy place. Desk risk lives in the hard place. Vision, language, medicine, genomics keep rediscovering this. Finance has the same disease with worse consequences, because the shifted world is where the P&L lives.

Research gap to name: predictive multiplicity metrics (disagreement rates across the Rashomon set; “tethered hacking intervals” for the range of a statistic across near-winners) exist so you can measure fan-out instead of hoping the champion seed is unique. Crowning one model without measuring the set is how arbitrariness becomes policy.

How this shows up in production

1. The illusion of a unique model. A desk trains twenty seeds for an unlisted structured claim that has never traded. Eighteen pass the same holdout band. The team celebrates “the model.” There is no the model. There is a cloud of near-winners. Picking one seed because its ID is lowest, or because a chart looked pretty, is a human lottery dressed as engineering.

2. The arbitrage trap inside one firm. Two agents can both be locally credible members of that cloud. One lives in a spread frame (marks from rate gaps) and lands near 1.50. One lives in a curve frame (marks from yield-curve shape) and lands near 2.10. Both passed training. Both can sound careful. If they are allowed to act on the gap, you may be trading measurement multiplicity, not opportunity. Bertrand named this across rulers. Underspecification says it survives inside one ruler when learning still leaves a crowd.

3. Credibility collapse under shift. Models that looked interchangeable on quiet months can separate when liquidity thins or the structure behaves oddly. Same training success. Different deployment selves. If your monitoring only watches average error, you learn about the break after capital already moved.

4. Midpoint laundering of the symptoms. When disagreement finally becomes visible, someone averages 1.50 and 2.10 into 1.80 and calls it compromise. That number was never in either family. It is a social object wearing a decimal. Underspecification does not get healed by averaging the symptoms.

CEH-001: two families, widths, and the fake peace

Keep the never-traded structure. Two desks can both be serious and still live in different frames.

A spread is a gap between two rates, issuer vs benchmark, or one tenor vs another. Spread-frame work treats those gaps as the main ruler for the mark. A yield curve lines yields up across maturities. Curve-frame work treats continuous curve metrics, level, slope, shape, as the ruler instead of raw spreads.

Same claim. Different geometry. That is why one honest family clusters near 1.50 and the other near 2.10. Now refuse the fantasy that each frame has one model. Look inside each family after honest re-seeding (five near-winners each, all clear the same holdout board):

Read that slowly. Width 0.19 and 0.17 are not mysticism. They are high minus low inside one allowed frame. Even after you lock a ruler, each frame can still hand you a band, not a point. Across frames the bands may not even touch: the spread cloud tops out near 1.61 and the curve cloud starts near 2.01 (empty measurement space in between). A production system that prints one float and a green checkmark has thrown away the only information that made the risk legible: the AI price was never unique.

This is why “we trained it” is not a risk answer. Training answered a different question: can some recipe meet the board. The desk question is: given everything that still meets the board, how wide is the decision, and do allowed frames overlap enough to act.

What breaks in the trenches

• Model trench: single-seed registry culture; no storage of near-winner set; no decision-object scores.
• Agent loop trench: “pick the best model” tool that returns one champion and discards siblings.
• Harness trench: only average loss / accuracy gates; no hard-slice pack; no fan-out metric on the mark.
• Pipeline trench: model cards without multiplicity fields; silent midpoint fields in the ticket schema.

What to lock before crowning a champion

• Treat near-winners as a set, not a single registry file.
• Score disagreement on the decision that spends, not only on average loss.
• Probe the hard slice, not only the quiet holdout.
• Make width and frame gap first-class risk objects before any auto-spend.
• Refuse silent midpoints that sit in no cloud (1.80 on CEH-001).

Building reliable AI infrastructure is less about crowning a champion faster and more about measuring whether the exam still admits many answers where money moves. The Harness is the gate that makes that crowd visible and unspendable when it is too wide or disjoint.

Book: Chapter 2 in Rulers Before Models.