I keep hearing timestamp your features as if that sentence finished the information problem. It does not.
They fixed the obvious time-travel joins. Labels no longer peek at end-of-day revisions by accident. Dashboards say as-of parity is green. Then a model still conditions on information it would not have been allowed to know at decision time. Not always through a future timestamp. Sometimes through a batch statistic, an agent memory, a normalized field fit on the full week, or a tool default that returns latest while the job claimed 14:00.
This is filtration as architecture law, not only as data hygiene. The information you are allowed to use is indexed by time and by role. If the predictor conditions outside that room, the exam was cheated even when the table looks tidy.
I keep running into the same sentence in reviews: we are point-in-time. Then someone opens an agent trace and latest is everywhere. Or a training job fits scalers on the full fold. Or a human mark made at 16:00 is treated as a 14:00 label because a person signed it. Seniority is not a filtration.
Quick definition, because filtration sounds like a lecture word
Think of a growing room of facts. At 14:00 the room contains what the firm could know at 14:00. At 14:10 the room is larger. A mark claimed at 14:00 is a claim about the 14:00 room. Conditional expectation in plain language is: the best prediction you can make using only the facts inside the allowed room. The tower idea is equally plain: if you predict using a bigger room, then try to restate that prediction in a smaller room, you should not magically keep the illegal advantage. If you do, the architecture leaked.
Illegal conditioning is any path that lets the model or agent use facts outside the room the job declared. Clean column names do not prove a clean room.
How the cheat survives clean tables
1. Soft leakage through aggregates. A normalization fit on the whole training week can encode future scale. A peer-group z-score can encode outcomes from names that printed later. The row timestamp looks fine. The statistic does not.
2. Agent memory as a second filtration. The agent remembers a curve print from a prior tool call at 14:20, then answers a 14:00 job with fluent confidence. Rights tokens cannot see the memory unless memory is scoped. Audit logs store the prose. They do not restore the illegal fact trail unless you designed for it.
3. Label sanctity theater. Humans mark after seeing afternoon revisions, then those marks re-enter training as if they were 14:00 truths. The model learns the revised world while production must speak in the earlier room. Research notebooks glow. Live parity dies, and people blame regime change.
4. Latest defaults. Tools love latest. Latest is not as-of. Latest is a hand in the cookie jar with a helpful API name.
5. Cross-job contamination. A research job legally uses a wider room. Its embeddings or distilled scores quietly enter the capital job. The capital contract still says 14:00. The features disagree.
CEH-001 under two rooms
At 14:00, dual frames still disagree: about 1.50 in spreads, about 2.10 on the curve view, gap near 0.60. At 14:20 a revision arrives that softens the conflict in hindsight. A research join quietly trains on the softened world. The production path, pinned to 14:00, still sees the conflict.
Promotion then compares a notebook hero to a production candidate and calls the gap model quality. It is often room quality. The notebook sat a different exam. More parameters will not reconcile two filtrations.
Worse, an agent asked to explain the gap fetches latest, invents a midpoint story near 1.80, and sounds careful while using a room the job forbade. That is not reasoning. That is illegal conditioning with better grammar.
Once you see it, you start noticing how many evaluation wins are really room wins. The notebook had more facts. The service had the contract. Calling that a model quality gap is how teams keep training on the wrong exam while production takes the blame.
Why this is deeper than feature-store slogans
Point-in-time stores help. They do not define the mathematical object the model is approximating. If your architecture can see a bigger information room than the decision is allowed to use, you built a cheat-capable machine. Adapted models, scoped memory, and tower tests are how you make the law executable. Without them, as-of becomes a sticker on a pipeline that still peeks.
I also stop trusting human labels as sacred by default. A human can sit tomorrow's exam too. If the label was born outside the room, training on it is still illegal conditioning wearing seniority.
What I take from this as systems work
I do not trust parity greens until someone can name the allowed information room in engineer language, poison-test the recipe with illegal fields, prove research and service agree when forced into the same room, and scope agent memory so yesterday's 14:20 fact cannot answer today's 14:00 job.
There is a systems smell I now treat as decisive. Research and production disagree, and the first explanation offered is model architecture. Before I accept that, I ask whether they shared a room. If the notebook saw afternoon revisions, peer stats fit on the week, or an agent memory of latest, architecture is not the first suspect. Filtration is.
That reordering matters. You can burn months swapping networks while both sides keep sitting different exams. More capacity makes the illegal room more eloquent. It does not make it legal. For CEH-001, eloquence around a softened 1.80 story is often just a bigger room talking.
Curious how others detect contamination that is not a simple future timestamp, especially in agent memory and batch normalization paths.