The naive answer - "train on the same box that serves" - works at small scale but breaks the moment training cycles cross the threshold where serving must be paused. The disciplined answer is to burst training to rented cloud GPUs while serving stays local. The economics are surprisingly favourable; the operational discipline is the part most teams underestimate. The output of a specialist programme is promoted improvements per quarter. Two variables drive this output:
Figure 1. Concept map for this appendix entry: fight, desk translation, what to lock.
The cycle-cadence-times-quality frontier
The output of a specialist programme is promoted improvements per quarter. Two variables drive this output: - Cadence - how often a training cycle runs. Weekly is twelve cycles per quarter; monthly is three. - Quality - what fraction of cycles produce a candidate that beats the gate. A mature corpus and a stable training recipe push this above 50%; an early or churning programme is lower. The product of cadence and quality is what actually ships: a weekly cycle at modest quality beats a monthly cycle at high quality, because cadence tends to dominate.
The cost of a single fine-tune: a worked example
Take a 7B specialist undergoing parameter-efficient fine-tuning on a corpus of a few tens of thousands of examples. The narrowest spot bet - a 4090 for one cycle - costs about a dollar and a half. A budget of $15 per month buys roughly ten cycles. For most specialist programmes, that is enough cadence to run weekly with two emergency cycles in reserve.
Spot vs on-demand: when the discount pays back the risk
Spot instances are 60-80% cheaper than on-demand but can be preempted by the cloud provider at short notice. For training, this trade-off is almost always in the buyer's favour because: 1. Training checkpoints. Modern training writes a checkpoint every few hundred steps; resuming from a preemption costs the time since the last checkpoint, not the time since the cycle started. 2. Preemption probability is bounded. At the 4090 / 3090 tier, preemption rates are typically only a few percent over a four-hour window, so the expected wasted compute is a few minutes per cycle. 3. The price gap is large - spot pricing runs 60-80% cheaper than on-demand, wide enough that the buyer can afford to get preempted more than once and still come out ahead.
The reversibility test: can you go back to local-only in one command?
The most important property of any cloud-burst integration is reversibility. The burst should be additive - it accelerates training when active, and the platform reverts to local-only operation when not. A burst whose absence breaks the platform is no longer a burst; it is a dependency.
Worked picture
A team has a $15/month cloud budget. A weekly fine-tune cycle costs $1.50 on spot or $4.50 on on-demand. - Spot strategy: 10 cycles × $1.50 = $15. Headroom for two extra cycles if budget allows. - On-demand strategy: 3 cycles × $4.50 = $13.50. Two cycles per month sit on the table unused. The spot strategy buys more than 3× the cadence for the same budget. The preemption risk costs maybe 5% of cycles being retried; the net is still ~9.5 cycles/month vs 3. The right answer is overwhelming. The on-demand strategy is the correct choice only if a single cycle's completion is itself deadline-critical - which is rare in a continuous-improvement programme.
What breaks in production
A subtle failure mode: the team comes to rely on the cloud for eval as well as training, then for part of the production serving path (a fallback that becomes the primary path), then for the embedding store - and one day the cloud is unreachable and the platform is down. The fix: bound the cloud's role in the architecture document. Cloud is for training and for explicit, documented serving fallback. Anything else - eval, embeddings, primary serving - stays local by design. The architecture must answer the question "what does the platform do when the cloud is gone?" with a clean and rehearsed "it continues to work."
Key idea
Cloud-burst training is the lever that converts a constrained local stack into a high-cadence improvement programme. The economics are favourable - a dollar and a half per cycle, on a budget of fifteen dollars per month, buys weekly cycles. The discipline is reversibility - the cloud is additive, never required. A platform that respects both ends up shipping four times more improvements per year without giving up the sovereignty that motivated self-hosting in the first place.