Evaluation is a measurement; promotion is an act with consequences. A promotion gate that runs only after training catches regressions but does not prevent the wasted compute of training on a stale or thin corpus. The preflight check runs before training begins and answers two questions:
Figure 1. Concept map for this episode: fight, desk translation, what to lock.
The preflight check: what must be true before training
A promotion gate that runs only after training catches regressions but does not prevent the wasted compute of training on a stale or thin corpus. The preflight check runs before training begins and answers two questions: 1. Is the training corpus ready? A small manifest file declares the corpus ready only after the build script has verified that it contains enough recent items, that no item has been duplicated from the eval pack, and that the schema is intact. If it isn't ready, training is refused with a clear error. 2. Is the held-out pack big enough? The eval pack carries its own manifest with an item count; if the count is below the threshold, the same refusal applies before any compute is spent.
The audit log: every promotion is one append-only line
Every promotion attempt - successful or rejected - writes one row to an append-only log. The log is never garbage-collected. It is the platform's institutional memory of which model served which user when.
Rollback: the one-command discipline
A promotion that cannot be reversed is not a promotion; it is a deployment. The discipline: every promotion produces, as a side effect, a rollback command stored in the audit log that, when executed, restores the prior state.
Shadow alias serving and A/B testing
The most aggressive promotion approach - "the candidate beats the gate; promote it; live with the consequences" - works for well-scored candidates on well-trusted gates. For higher-stakes upgrades, a shadow phase reduces risk. In a shadow phase, the candidate is served as a parallel alias to the production alias. Production calls hit both - the production alias's response is returned to the user; the candidate alias's response is logged but not surfaced. Over a few days, the engineering team reviews the disagreements: where do the two answers differ substantively, and are the differences improvements or regressions? Only after that review accepts the candidate does the alias map get updated - the candidate becomes production, and the prior production becomes the rollback target. For lower-stakes upgrades, the shadow phase is overkill and the gate alone is sufficient; the judgement of when to use one is operator-side, weighted toward high-traffic and customer-facing surfaces.
Definition
A tier-stratified promotion gate is a promotion gate whose thresholds vary by model tier (fast, workhorse, heavy, judge, …) according to each tier's natural cost-quality trade-off, declared in a single shared manifest. The same generic gate runner applies the appropriate thresholds for the tier of the model under test, so adding a tier is a manifest edit and not a code change. Tier-stratified gates prevent the simultaneous failure modes of gating too tightly on cheap tiers (refusing all promotions) and gating too loosely on expensive tiers (accepting hidden regressions).
Worked picture
A candidate cycle runs without preflight. Training takes six hours. The eval gate runs; the candidate appears to beat baseline by a few points. Promotion completes; production ships the new model. Two days later, the operator notices that the eval pack contained only 38 items - below the statistical-power threshold. The improvement is within noise. Worse: investigation reveals that twelve of those 38 items had also been added, last week, to the training corpus by a sync script that should not have done so. The candidate had seen a third of the eval set during training. The improvement is meaningless. Rollback is executed. Future cycles run with preflight enforced; the same cycle never repeats. The cost of the omission: one bad model in production for two days, plus the trust cost of the operator wondering for weeks afterwards whether the next promotion is also tainted. The preflight check is cheap insurance against this exact failure mode.
What breaks in production
A subtle failure: an operator, in the middle of an incident, manually edits the config file to "put the old model back" instead of running the canonical rollback command. The file is changed but the audit log records nothing - there is no row, no commit, no reasoning. A week later, the operator wonders "why is the production model not the one we just promoted?" and re-runs the original promotion. The cycle repeats. The audit log shows one promote, no rollback, one promote - the rollback simply does not exist in the institutional record. The discipline: rollback is a first-class action with its own log row. The canonical rollback script writes the row. Manual edits that bypass the script are a process bug to be caught in retrospectives, not a normal mode of operation.
Key idea
The promotion gate is the governance layer of continuous fine-tuning. Preflight checks block wasted compute. The audit log preserves institutional memory. The one-command rollback turns incidents from forty-minute scrambles into five-minute fixes. The shadow phase catches the quality dimensions that no metric can measure. Tier-stratified thresholds make the gate honest across a fleet of differently-sized models, refusing the simultaneous failure modes of over-strict gates on fast tiers and over-loose gates on heavy ones. Skip any of these and the system runs faster - until the day it ships a regression that takes a week to recover from. Keep all of them and the system upgrades itself, weekly, without drama.