A promotion score is not a measurement until it carries a delta, an uncertainty, and enough sample to see the effect you care about. The slide says eighty-four percent. Or plus two points versus last week’s champion. There is a green check. Someone writes promote. Capital starts trusting a point estimate the way it trusts a voltmeter. That habit is older than modern agents. It is also how noise gets a job title.
What a claim owes you that a point score does not
A claim like “this model is better on CEH-001 marks” is a statement about a population, not about one lucky draw. It owes three objects:
- Delta: the paired difference on a frozen decision object (mark error, hedge residual, dual-frame honesty), challenger minus champion on the same cases.
- Uncertainty: an interval around that delta (bootstrap under block structure is the desk default).
- Power / MDE: the minimum detectable effect the desk pre-commits to caring about, and whether the pack can see it.
Minimum detectable effect (MDE) is policy written before unblinding, not a post-hoc story. Power is the probability that an honest exam would detect an effect of size MDE if it were true. An underpowered pack is a coin flip with a spreadsheet. Chebyshev-style concentration is the plain version of the sample-size point: if you want a tight claim about how far an average can wander, you need mass honest enough for the claim. Rare failures need more evidence than common ones. Thirty quiet days without a miss do not certify a tail.
Promotion is a rights change: tiny size, auto allow, fewer escalations. If the rights change rides on an unpowered score, you did not validate. You held a ceremony. Once a point score becomes the language of go or no-go, every incentive points at moving that number. People retune features, prompts, and pack membership until the cell turns green. Without power and freeze rules, that is specification search (researcher degrees of freedom; Simmons, Nelson, and Simonsohn’s “researcher degrees of freedom” framing), and finance has paid for it under every other brand for decades.
Why scoreboard culture feels like science
Numbers end meetings. Confidence intervals prolong them. Best-of-N search looks like diligence: try twelve prompts, twelve seeds, twelve feature recipes, keep the winner. Multiple comparisons hide inside that diligence. Under a complete null with nine independent one-sided five-percent tests, the chance that at least one sibling clears by luck is \(1 - 0.95^{9} \approx 0.37\). Selection without correction inflates wins. The champion is not only the best model; it is the luckiest survivor of your search. Family-wise control (Holm 1979 step-down) or false-discovery control (Benjamini and Hochberg 1995) are the classical corrections. Uncorrected best-of-N is theater.
Holdout packs that stay unlocked while candidates evolve are another quiet cheat. If the exam can be retuned against, the score stops being an exam. Optional stopping (peek, retune, stop when the story looks good) is the same sin under another name. Effect-size blindness completes the costume: a two-point lift can be real and still too small to matter for desk risk, or too small to detect reliably with the pack you own. Cohen’s power arithmetic is not academic fussiness. It is asking whether your ruler can see the notch you claim to have cut.
Underspecification makes the point sharper at production scale. D’Amour et al. (arXiv:2011.03395) show that many models can match a training objective while disagreeing on the decisions that matter. A green holdout cell does not mean the decision object was unique, powered, or hard-slice honest. On CEH-001 the decision object is dual-frame mark quality and midpoint refusal, not a free-floating accuracy percentage.
How this shows up in production
- Promotion by green cell. Dashboard shows champion ahead by two points. No interval. No paired view. No statement of decision-relevant lift. Gate opens.
- Best-of-N inflation. Twelve variants, one winner, one story. Nobody publishes sibling count. Nobody runs Holm or BH. The slide win rate is not the fresh-pack win rate.
- Calm-pack theater. Eval set is last quarter’s quiet weeks. Hard co-break weeks are under-represented. Live stress flips the ranking.
- Rare-event slogans without teeth. “We never saw a dual-frame collapse on the eval pack” becomes a safety claim when the pack never contained the collapse mode.
- Binary pressure. The meeting only knows yes and no. Underpowered wins must become yes. The third honest exit, inconclusive, never enters the room.
CEH-001: the unpowered plus-two that wants to spend
Keep the never-traded structure. Spread-frame work still lives near 1.50 [1.42, 1.58]. Curve-frame work still lives near 2.10 [2.01, 2.22]. The desk cares about mark quality under both frames, crowd width, and whether a candidate invents 1.80 peace.
Candidate B beats champion A by two points on a calm mean-error pack of twelve day-blocks. Slide says promote. Look closer:
| Object | Value | Read |
|---|---|---|
| Calm pack size | 12 day-blocks | Frozen case list (toy) |
| Point delta (B − A) | +2 points | Slide story |
| Bootstrap CI on delta | [−4, +8] | Zero inside; noise-compatible |
| Pre-registered MDE | 5 points | Smallest lift that changes rights |
| SE (block noise) | ~3 points | σ / √n scale |
| Power vs MDE=5 | ~50% | Coin flip registered as science |
| Mass for ~80% power | ~36 blocks | Not twelve |
| Hard-pack ranking | A quieter | Co-break / missing-curve weeks |
| CEH dual marks | 1.50 / 2.10 | Still dual; deny 1.80 |
| Frame gap | 0.60 | Not healed by a green cell |
The two-point win is compatible with no real lift, and it sits under the MDE. On a harder pack with co-breaks and missing-curve days, the ranking flips: A is quieter and less wrong on the weeks that hurt. Dual-frame disagreement is still wide. B is slightly better at sounding settled near a midpoint culture the desk already mistrusts.
Notice what the two-point story erased: whether the same cases drove the win; whether nine siblings were discarded; whether hard weeks were in the room; whether a lift that small should change autonomy at all. Erasure is the product. The percentage is just the packaging. Inventory of 40,000 units still turns a 0.60 dual-frame gap into real money; an unpowered champion does not make that gap honest.
What breaks in the trenches
- Model trench: training loss never sees pack power; optimizer happily overfits a calm CSV.
- Agent loop trench: “pick the best model” tool returns a best-of-N survivor with no search card.
- Harness trench: CI has a percentage cell and no paired bootstrap, no MDE field, no hard-slice gate.
- Pipeline trench: promote is a merge button; inconclusive is not a state machine exit that blocks spend.
What to lock before automation
- Frozen exam hash (cases, seeds, frame tags, hard-slice membership) before candidates compete.
- Paired delta plus interval on every promotion claim.
- MDE and power note at registration; refuse lifts below MDE.
- Search card with sibling count and Holm or BH correction.
- Hard packs that can flip a calm winner; three-way exit including inconclusive.
- CEH-001 dual marks stay dual; 1.80 never becomes a promote label.
Problem: unpowered point scores and uncorrected search turn promotion into noise theater. Solution lives in the next chapter: a harness that can refuse a noisy win. Until then, treat eighty-four percent and plus-two as costumes unless delta, interval, and power show up on the same ticket.
What breaks in the trenches without the harness
- Model trench: no exam_hash binding on train logs; research quietly edits pack membership mid-sweep.
- Agent loop trench: promote tool treats green cell as allow; inconclusive is not a typed exit.
- Harness trench: evaluate step is a report that cannot block merge; search card is a wiki page.
- Pipeline trench: calendar pressure renames inconclusive to “watch and ship.”
Book: Chapter 8 in Rulers Before Models.