Today's exam, not tomorrow's answer key

Like a short physical law: a mark is a claim about an information set at a clock time. Hand the model anything from after that clock, and the exam was open-book no matter how green the gate looks.

I keep finding models that look brilliant in research and fall over in production for a boring reason. They sat today's exam with tomorrow's answer key. A second cheat rides with it: stress packs that turn one dial at a time and call the book resilient while real breaks arrive as bundles.

These cheats are less flashy than Bertrand. They wreck more production systems. They survive the posts people like to share, the ruler talk, the crowd talk, the swagger talk, and hide in joins and stress catalogs. On CEH-001, the never-traded nameplate where one frame keeps saying about 1.50 and another about 2.10, both cheats invent a story of readiness that live marking cannot keep.

Quick definitions before the trenches

As-of (point-in-time): every capital-touching feature, tool fetch, and label is legal only if it was knowable at the decision clock t, not after it. Industry language is the same: point-in-time correct joins so training data reflects values available when the label was observed, not "latest."

Look-ahead / temporal leakage: any ancestor of a feature (or of a training label) with knowledge time after t. In finance it is the same family as classic look-ahead bias: the model learns from information it would not have had when the decision had to be made. Recent work on LLM sentiment backtests (Glasserman and Lin, arXiv:2309.17322) restates the same wound for modern tools: if training knowledge overlaps the evaluation window, the exam was open-book.

Bitemporal honesty: store both valid time (when the fact was true in the market) and knowledge time (when the firm could have known it). A revised curve published at 14:20 that corrects a 14:00 print is not "14:00 data" under knowledge time. Bitemporal stores exist so backtests cannot silently promote revisions into the past.

One-at-a-time stress: shock spread alone, then rates alone, then liquidity alone, then declare resilience. Co-break (joint bundle): the day those dials move together. Separate columns are not independence.

Cheat one: the answer key arrived early

A mark at 14:00 is a claim about what the firm knew at 14:00. If a feature silently includes a 14:10 print, a revised curve published at 14:20, or a label only knowable at day's end, the model sat the exam with the answer key. Training metrics soar. Demo day glows. Live marking at 14:00 with only 14:00 information falls on its face, and people blame regime change instead of time travel.

On CEH-001 the tell is ugly and specific. Dual honest bands still sit near 1.50 [1.42, 1.58] and 2.10 [2.01, 2.22]. The research notebook matches the afternoon revision almost perfectly, sometimes lands near the diplomatic 1.80 that the dual-frame card forbids. The production path, pinned to 14:00 snapshots, disagrees with both the notebook and the other frame. Teams then "fix the model" when they should fix the clock. The model was never the genius. The join was.

The painful part is how virtuous the leak feels. Someone added a better feature. Accuracy rose. A dashboard got greener. Nobody asked whether the feature was legal at decision time. In financial AI, illegal information is not a pedantic rule. It is a different exam. Point-in-time feature stores exist for exactly this reason: a "get latest" query is fine for batch scoring and catastrophic for historical training sets. Latest is a hand in the cookie jar.

Soft leakage: four rooms, tidy stamps

Timestamp hygiene is necessary and not sufficient. Four rooms operators actually build, and only the first is a future-timestamp canary:

1. Timestamp room. Prints with knowledge time ≤ t only. Tomorrow's 10y is the hard canary.
2. Aggregate room. Peer z-scores, scalers, PCA loadings fit only on data legal at t. A full-week fit applied to a Thursday 14:00 mark has no future stamp in any column, aggregates carry no timestamps of their own, and still cheats. The feature can flip sign under the illegal window.
3. Memory room. Agent scratchpads and tool defaults that return "latest," including a 14:20 curve revision on a 14:00 job. Feature stamps look tidy; the content is not.
4. Label room. End-of-day revised marks as supervision for a midday decision teach the answer key. The feature store can look clean while the label still time-travels.

That is why "as-of green" on a column dashboard is not a Clearance artifact. The contract must hash recipes and windows, not only rows. Bitemporal knowledge time is the check that catches the revision that rewrote yesterday without changing any event-time stamp.

Cheat two: separate shocks, independent fantasy

Desks often shock spread, then shock rates, then shock liquidity, each in isolation, then declare the book resilient. Real breaks arrive as bundles. Spreads gap while funding tightens while quotes go missing. If your AI only saw one-at-a-time pain, its crowd width and ranges are optimism machines.

"We modeled them as separate" is not the same claim as "they are independent." Independence is a strong modeling choice. Most market distress laughs at it. On CEH-001, spread-frame work near 1.50 and curve-frame work near 2.10 can keep a familiar calm gap near 0.60 under single knobs. Under a named joint co-break, equity features and the discount pack moving together, the gap can widen toward something uglier (~0.95 in the numbers we keep using), while overlap stays empty. Both clouds jump; neither becomes the midpoint 1.80.

Factorized pipelines encourage the fantasy. Each service owns one driver. Each dashboard panel looks fine. Nobody owns the day when the drivers move together. That day is when the mark that spends either becomes honest or becomes an accident.

How this shows up when people trust the green dashboard

1. Notebook versus service parity theater. Offline eval used a join production cannot reproduce at as-of time. Offline looks calibrated. Online is a different animal wearing the same model name. Training-serving skew is the industry name; treat it as a release blocker, not a wiki page.

2. Agent fetch discipline failure. The agent helpfully pulls the newest curve because the tool default said so. Clearance sees strong ranges built on illegal information and allows. The allow was logically consistent and economically cheated.

3. Stress packs that never co-break. Hard-slice probes only perturb one factor, so you get false interchangeability across near-winners. Joint stress is where multiplicity and dependence show their teeth together.

4. False comfort from marginal intervals. Spread range looks fine. Curve range looks fine. The joint scenario that moves both is never generated, so overlap rules never see the day that matters. Green panels, empty overlap under the only stress that was real.

5. Leakage manufactures the midpoint. On paths where the future print dominated the residual, a leaked model can print exactly 1.80 with zero exam error while the honest dual-frame mark carries real residual. The leak did not only inflate accuracy. It manufactured the peace number the card forbids and then scored perfectly for doing so.

Walk the failure on familiar numbers

The thought process I want on the desk is suspicious by default: what clock did this feature obey, and which other clocks moved with it when the world got ugly. If nobody can answer, the earlier gates are decorating a leak. More model capacity will not invent a clock. A better prompt will not invent a joint.

Dependence fantasy has a dashboard cousin too. People show a correlation heatmap from calm months and declare the joint understood. Calm correlation is not co-break behavior. The bundle that matters shows up when liquidity is already thin and both frames are moving.

What breaks in the trenches

• Model trench: labels or features with knowledge time after t; scaler windows fit on the full sample including the future.
• Agent loop trench: tool defaults return "latest"; scratchpad carries a post-clock revision into a 14:00 job.
• Harness trench: only one-at-a-time stress; green marginal intervals; no co-break width object.
• Pipeline trench: notebook/service parity never tested at the job clock; bitemporal knowledge time not stored; Clearance binds a snapshot live cannot rebuild.

Feature / legal slice (information contract at clock t)

Book Ch 16 (research and data flows under an information contract) names the engineering object: an information contract is the legal slice of the world at decision clock t. Call the capital-touching row set a feature / legal slice: sources, tenors, as-of stamps, recipe hash, owner, refuse on leak. Not a product-management “vertical slice.” A feature store slogan without these fields is still open-book exam risk.

Dual honest marks (~1.50 / ~2.10) computed under a leaked pack are not dual marks. They are open-book theater. Leakage is never an excuse to ship 1.80; it is a hard refuse before geometry is even discussed.

Convention is a ruler too: three worked leaks

Everything above is about which row is legal at the decision clock. A separate, quieter class of leak lives in which recipe was legal, and it survives a perfect timestamp audit because conventions and bootstrap inputs carry no clock of their own. Three fixtures, all traceable to real dollars on CEH-001.

The pasted convention. The vendor prints a ten-year par yield of a semi-annually compounded 4%. The frame card locks continuous discounting. One desk computes the honest continuous equivalent and gets a discount factor of 0.6730. Another desk, still claiming the continuous frame, silently pastes the vendor's 4% as if it were already continuous: discount factor 0.6703. The gap in the discount factor looks tiny, but on a curve-implied distress recipe that maps discount levels into a mark, it is enough to nudge the whole crowd: the honest recipe places the curve cloud near 2.10, the naive paste drifts it toward 2.04. Both numbers are, in plain English, "continuous discounts of a four-percent vendor yield." Only one matches the card's conversion hash. Averaging 2.10 and 2.04 into 2.07 is midpoint laundering wearing a rates costume, the exact same sin as blending spread and curve frames.

The day-count that isn't formatting. Take a 182-day accrual period at a quoted 4%. Under ACT/360 the period accrual is 182/360 × 4% = 2.0222%. Under ACT/365 it's 182/365 × 4% = 1.9945%. The difference is 2.77 basis points of the period — on a $50 million notional leg, that's $13,850, arising entirely from which of two conventions a downstream job assumed. Neither convention is wrong; both get used, by the same vendors, on the same day, for different instruments. A curve pack whose instruments carry no day_count_id cannot be hashed into a legal recipe, because the same quotes under two bases produce two different curves, and a hash over quotes alone can't tell them apart.

The bootstrap that leaked tomorrow's bond. Bootstrap the curve legally: a 6-month bill at 0.9800, a 1-year bill at 0.9615, and a 2-year coupon bond (face 100, coupon 4%, dirty price 100.50) all known by the clock. Solving gives B(t,2) = 0.929365, a two-year spot of 3.7306%, and a one-year-forward-one-year rate f(1,2) = 3.4577%. Now let the join insert the 2-year bond's print from the next morning, dirty price 100.95, into today's strip. Redone with the leaked price: B(t,2) = 0.933692, spot 3.4899%, forward 2.9783%. The two-year spot moved 24.1 basis points; the one-year forward moved 47.9, nearly double, because a forward is a ratio of two discount factors, so an error in the longer factor gets levered by the shorter one, not diluted by it. With this desk's declared key-rate sensitivity of 0.0042 mark units per basis point of that forward, the leak shifts the curve-frame proposal by 0.0042 × 47.9 ≈ 0.20, from an honest 2.10 to a contaminated 2.30.

Three consequences follow from that last fixture, and the third is the one to remember. The dual-frame gap appears to widen from 0.60 to 0.80, which a desk watching gap width as a health metric would misread as increased market disagreement — exactly backwards. The contaminated mark is actually more confident, not less: leaked features fit better, the conformal band narrows, and the leak looks like a modelling improvement. And the peace number moves too: the naive midpoint of 1.50 and the contaminated 2.30 is 1.90, not 1.80. A desk that had negotiated its way to 1.80 as the number everyone could live with would find that number quietly following a contamination nobody detected. The honest marks each carry a recipe whose sensitivity to every input is documented, so when an input is wrong you can trace it. The midpoint has no recipe. When it moves, nobody can say why.

What to lock before automation

• Treat as-of as a hard contract on every capital-touching feature, tool fetch, and label, not a nice-to-have join hygiene note.
• Hash recipes and windows (bootstrap inputs, scaler fit windows, tool as-of arguments), not only data rows. Prefer bitemporal knowledge time where revisions rewrite history.
• Pin agent tools to the job clock; quarantine default-latest to research-only paths with no spend tokens.
• Fail builds when notebook and service disagree under replay at the job clock (training-serving skew as release blocker).
• Re-score crowds and ranges on named co-break bundles; let Clearance consume co-break width, not only calm holdout smiles.
• On CEH-001: dual marks stay dual. Leakage is never an excuse to ship 1.80.

Building reliable AI infrastructure, at least for me, is less about a smarter learner and more about refusing open-book exams and one-knob courage.

Curious how others catch as-of leaks in agent tool calls before a parity incident, and whether their stress harnesses can name joint bundles rather than only single knobs.

CEH-001 under two rooms

At 14:00 the dual-frame research marks still sit near 1.50 [1.42, 1.58] and 2.10 [2.01, 2.22]. A notebook at 14:20, after a late curve print and a batch rescale, looks reconciled near 1.80. Both sides call it model disagreement. It is not. It is two exams. One room was legal. One room inherited someone else's afternoon. Eloquence around a softened 1.80 story is often just a bigger room talking.

Work the aggregate-room version of the same cheat on real numbers, because "the scaler was fit on the full week" sounds like a technicality until you see what it does to one name's feature value. CEH-001's raw spread level, 1.50, gets standardised against a peer group before it enters scoring. Fit the peer mean and standard deviation on the full Monday-to-Friday window (mean 1.94, sd 0.42, both legal-looking timestamps since aggregates carry no timestamp of their own) and the raw level becomes a z-score of -1.048. Fit only on what was legal by Thursday 14:00 (mean 1.72, sd 0.51) and the same raw level becomes -0.431. The illegal scaler makes the name look about 2.43x more extreme than the legal one does — not a rounding difference, a name that clears a "-1 sd" screening threshold under one window and clears it under neither timestamp check, because there is no future-dated column to catch it. The two scalers do not disagree about the market. They disagree about which week the desk was allowed to know about.

Note where the distortion comes from: the legal window has a lower mean and a higher standard deviation, because Friday's prints happened to be calm and tight, and fitting on the full week imports exactly the knowledge a Thursday 14:00 decision does not have. Replay the co-break stress with the legal scaler alone (equity and rates moving together on a shared funding factor) and the honest frame gap widens toward ~0.95, not the 0.60 calm-world figure — the model was never as good as the leaked exam said, and the exam was never measuring what it claimed.

Book: Chapter 6 in Rulers Before Models.