The problem we left open
In the last post I used underspecification to name a failure mode that keeps showing up after people think training finished the job.
You can clear the same holdout with many different predictors. On average error they look interchangeable. On the mark that moves money they can disagree. Champion culture hides that crowd. Midpoint culture invents a fake peace between 1.50 and 2.10 and calls 1.80 reconciliation.
That creates three real headaches in production:
- The illusion of a unique model: shipping one seed from a flat board as if the board proved uniqueness.
- The arbitrage trap: two credible near-winners, two frames, a gap that may be multiplicity rather than edge.
- Credibility collapse under shift: quiet agreement that falls apart when the hard week arrives.
So the question for this post is simple. If that is the failure mode, what does a real AI solution look like?
The solution, as one stack
The core idea is older than today's agent demos, and it still holds: stop asking which model won until you have asked how wide the set of models that still win is, on the decision that matters, under the stresses that matter.
I turn that into five moves. Together they are one solution, not five optional add-ons.
1. Probe the near-optimal set on purpose
Do not train once and file the artifact. Force a small, honest sample of near-winners into existence. Re-run with different seeds. Draw mild hyperparameter neighbors that still clear the board. Where retraining is expensive, use cheap stochastic passes at inference that shake the network enough to reveal instability without a full retrain.
You are not hunting a prettier champion. You are estimating whether good on the board is a point or a cloud. Ten to thirty near-winners often shows whether fan-out is tiny or dangerous. Exact count matters less than the rule: no production mark from a recipe that has never shown you its siblings.
If finance says we only have GPU for one run, then autonomy is not the right ambition yet. Scarcity of compute is not a reason to fake uniqueness.
2. Score fan-out on the decision that spends
Average error can be flat while marks move. Define the decision object up front: the mark in each allowed frame. For every near-winner, record that object. Publish simple width: high minus low inside the spread family, high minus low inside the curve family, and the gap between families.
This is where many uncertainty dashboards lie to themselves. They show loss variance or embedding drift and never show whether the mark moved by three cents or sixty. Capital does not care about your embedding plot. Capital cares whether 1.50 and 2.10 are both still legal winners of the same exam.
3. Add a hard-slice probe, not only the quiet holdout
Quiet months create fake agreement. Build a small stress pack for the structure: thinner liquidity, jump-like moves, missing quotes, regime tags the desk already fears. Re-score the same near-winner set on that pack.
If the cloud was narrow on the easy holdout and wide on the hard pack, interchangeability was an artifact of the exam. Keep the pack small enough that people actually run it. A dozen brutal scenarios that block release will change behavior. A hundred lovingly named ones that nobody executes will not.
4. Turn disagreement into a first-class risk object
Once width and frame gap exist as numbers, stop treating them as embarrassing noise. They are the risk signal.
Useful policy language looks like this: if within-frame width exceeds desk tolerance, deny auto-mark and escalate. If allowed frames do not overlap, deny auto-mark and escalate. If they overlap and width is inside tolerance, allow only inside the overlap region, not a single heroic float.
Notice what this replaces. It replaces pick the seed with highest confidence. It replaces average the agents. It replaces ship the registry champion. Those heuristics optimize for narrative closure. This policy optimizes for not spending when the near-optimal set still argues.
5. Make silent midpoints unable to spend
Whatever gate you use before blotter rights, its inputs must include crowd width and frame gap, not only a point estimate. Encode a hard forbid: no auto-posted mark that sits outside every near-winner cloud, including the classic 1.80 compromise between 1.50 and 2.10 families.
Escalation is not failure. Escalation is the system admitting the problem is still under-determined for autonomy. Humans can still choose a frame, widen tolerance, gather data, or refuse the deal. What they cannot do is let the runtime invent a peace number and call it a model output.
A concrete walkthrough
Same never-traded structure. Quick reminder of the rulers: spread-frame = mark from interest-rate gaps; curve-frame = mark from yield-curve metrics (yields by maturity). Probe five spread near-winners and five curve near-winners. All pass the same board.
Spread outputs: 1.42.. 1.61 width = 1.61 − 1.42 ≈ 0.19 (sibling disagreement inside one frame)
Curve outputs: 2.01.. 2.18 width = 2.18 − 2.01 ≈ 0.17
Hard-slice pack: widths stay wide; intervals still fail to overlap
Frame gap: centers about 2.10 − 1.50 = 0.60 (frame conflict, not width inside one family)
Policy:
width above tolerance? escalate
intervals overlap? no to escalate
proposed mark equals 1.80? deny (not in any cloud)
auto-allow region today? empty to no autonomy
The satisfying ending is not a cleverer point estimate. It is an explicit empty auto-allow region with a logged reason. That is what a researched solution means on a desk that wants to sleep: the machine stopped itself for a named cause, not because someone felt nervous.
The flow in one breath
locked frames
to probe near-winners
to score decision fan-out + hard-slice pack
to publish width + frame gap
to deny or escalate if too wide or disjoint
to never auto-spend a silent midpoint
That is the systems work I care about after underspecification. Curious how others store near-winner sets in real registries, and what tolerance language their gates enforce when two passing families refuse to overlap.