Like a short physical law: freeze the exam before the candidate. Choosing the test after you see the answers is not science, and it is not model risk either, it is how thin governance re-enters a thick-looking machine.
You can lock rulers, probe crowds, insist on stakeable ranges, split think from act, pin as-of clocks, name joint stress, budget paths, and cost hedges. Then Friday arrives. Someone changes the exam after they see the answers. A metric leaks in. A friendly subset replaces the hard week. A joint bundle gets quietly dropped. A confidence threshold gets tuned in the war room until the candidate looks green. That move feels like delivery.
This post is not new science. It is the ship discipline that keeps the science from being negotiated away. I treat it as a promotion runbook for a never-traded family I keep calling CEH-001: the structure where one frame keeps saying about 1.50 [1.42, 1.58] and another about 2.10 [2.01, 2.22], and the social temptation is always the peace number in the middle (1.80, denied).
Choosing the exam after the answers is not science
In research language this is pre-registration: lock the analysis plan before looking at the confirmatory sample. On a desk it is simpler. Freeze the exam before the candidate is trained for release. Metrics, path budgets, world cards, parity packs, honesty slices, co-break bundles, and policy versions get a release id. Candidates may fail. Humans may later decide the exam was wrong. Fine. Then change the exam under a new id, and the candidate re-enters as new. What you may not do is tune the test until the model passes and call that science.
A frozen exam is a folder with a hash and an owner. If the hash changes, the release id changes. That single inconvenience blocks a remarkable amount of self-deception. Challenger versus champion is allowed. Changing the scoreboard mid-fight is not. If the challenger needs a new metric to look good, that metric joins the next frozen exam, and both candidates run again. Banking model-risk practice (SR 11-7 lineage: independent validation, documentation, ongoing monitoring) is the same social technology under a different letterhead: the exam is not owned by the people who need the model to pass this week.
Same rule for worlds. Shopping a gentler World S′ after the challenger fails S-joint is selection inflation wearing a stress pack. Worlds are pre-registered on the card before the run; adding one mid-study requires a change packet and a re-registered exam.
One green check is how theater returns
A CEH-001 family release has to clear several gates that map to the headaches in this arc:
- Ruler and job contract present; dual-frame fields; midpoint forbid.
- Near-winner crowd probe published; ranges and coverage monitor healthy.
- Typed job schema valid; propose / Clearance / act rights split.
- As-of parity green; leakage canaries quiet (hard + soft rooms).
- Joint co-break bundles scored; path budgets held on two named worlds.
- Residual honesty clean; policy dry-run on replay.
One gate fails, the build fails. Averages across gates are forbidden. Weighted scorecards are how people waive the gate they fear. Publish the gate map next to the architecture diagram. If a gate has no owner, it will rot. If a gate can be skipped with a comment, it is not a gate. Comments belong in tickets that change the frozen exam, not in green builds. Gate owners should be people, not committees. Committees ratify. Owners get paged when a gate goes soft. Soft gates are incidents even when P&L is quiet.
Replay packs beat review-meeting eloquence
Arguments in promotion meetings are cheap. A replay pack is not: same jobs, same clocks, same worlds, same policies, new candidate versus champion. Diff the allow decisions and the act receipts. If you cannot show why allows changed, you do not promote.
Include known poison pills on purpose:
| Poison pill | Must still do | Maps to |
|---|---|---|
| Midpoint 1.80 attempt | deny (outside both bands) | Ch 4 frame card |
| Latest-fetch / post-clock curve | refuse or fail parity | Ch 6 as-of |
| Calm-only candidate under co-break | width toward ~0.95; empty overlap then escalate | Ch 16 joints |
| Friendly-world-only path budget | fail World S | Ch 12 worlds |
| Costless hedge, residual still predictive | fail residual honesty | Ch 7 hedge studio |
The pack should prove the gates still bite, not only that the new model looks clever on quiet days. Replay weekly even without a release. A pack that only runs under launch pressure will be under-maintained and over-negotiated.
Escalate-only until the theater dies
While CEH-001 still has dual-frame conflict, unfinished honesty, or unfinished path rulers, default posture is escalate-only for capital actions. Allow tokens from Clearance require:
- A typed job with dual frames and no midpoint field.
- Crowd width inside tolerance on calm and co-break.
- Overlapping stakeable ranges or an explicit human frame choice on the record.
- Coverage breakers green; as-of parity proven for this recipe.
- Hedge policies scoped to two passed worlds with costs on.
Missing artifact means escalate. Someone proposes 1.80 as peace means deny. Autonomy widens only after clean replay weeks, not after a demo. That is deliberately unheroic. It will frustrate people who wanted the agent to just handle it. That frustration is the point. Unfinished geometry is not a vibe. It is a stop condition.
Staged autonomy, not a flip switch
When gates stay green, widen in stages: propose-only, then draft-plus-human-decide, then allow on tiny size (canary), then size up inside token bounds. Each stage needs its own replay evidence. A single demo day cannot skip the ladder. Skipping the ladder is how fused think-and-act rights return wearing a launch announcement.
Canary does not mean silent. Canary means scoped tokens, loud monitoring, and a pre-agreed kill switch that does not require a meeting to fire. Meetings are how yellow dashboards become red P&L. Either risk co-owner can pull a lease alone; that is part of the runbook, not a special emergency.
CEH-001 promotion runbook v1 (one page)
| # | Lock | Refuse if missing |
|---|---|---|
| 1 | Freeze exam hash before candidate trains (metrics, worlds, parity, co-break, budgets, residual tests) | no release id |
| 2 | Multi-gate CI; fail closed on any single gate | weighted "overall green" |
| 3 | Replay pack vs champion; poison pills still bite | meeting eloquence only |
| 4 | Default Clearance: escalate-only on CEH-001 capital acts | auto-allow while dual conflict |
| 5 | Dual marks required; 1.80 deny; human frame choice if uniqueness needed | blend field / silent average |
| 6 | Shadow lease only after statistical + path + as-of gates pass | demo-day promotion |
| 7 | Canary size only after clean shadow weeks + kill switch live | skip ladder |
| 8 | Sized rights only after registered change packet + dual ownership | single-owner launch |
| 9 | Any exam change means new id; candidate re-enters as new | mid-fight scoreboard edit |
| 10 | Monday audit: full chain without chatting the model | transcript archaeology |
What breaks without promotion locks
- Model trench: candidate re-tuned against a moving exam; "green" is path-dependent on Friday edits.
- Agent loop trench: launch flips fused spend rights on; canary ladder skipped.
- Harness trench: one soft average across gates; poison pills not in CI; World S dropped quietly.
- Pipeline trench: no frozen exam hash; Monday cannot reconstruct which scoreboard the candidate passed.
What to lock before autonomy widens
- Pre-registered exam hash and owner; change packet for any edit.
- Multi-gate fail-closed map with human owners, not weighted scorecards.
- Replay pack with poison pills that still deny 1.80 and still catch leaks and world shopping.
- Escalate-only default while dual-frame conflict remains.
- Staged autonomy ladder with dual-owned kill switch.
What I take from this as systems work
Done is not a bigger model. Done is when you can point at a CEH-001 decision and replay the chain: which ruler, which crowd, which ranges, which clock, which joints, which worlds, which Clearance token, which act. If that chain is missing a link, the system should sound dull and refuse. Dull refusal is the product.
I care about this because every thick stack I have seen can still be defeated by one social move: renegotiate the exam after the answers. Frozen exams, multi-gate builds, replay packs, escalate-only defaults, and staged autonomy are how you stop that move without pretending science is optional.
Curious how others enforce frozen exams across research and production without turning every release into a negotiation about which gate to waive.
Book: Chapter 24 in Rulers Before Models.