This chapter designs a priority admission controller: two lanes, structured 503s, and a watchdog - the boring infrastructure that protects interactive latency. Classical queueing theory - Erlang, M/M/1, M/M/c - assumes that service times are independent of arrival times and that service is preemptable or at worst non-blocking. Neither assumption holds for local LLM inference.
Figure 1. Concept map for this appendix entry: fight, desk translation, what to lock.
Why LLM serving violates the assumptions of classical queueing theory
Classical queueing theory - Erlang, M/M/1, M/M/c - assumes that service times are independent of arrival times and that service is preemptable or at worst non-blocking. Neither assumption holds for local LLM inference. Service times depend on context length and on what is already loaded. A 200-token request to a warm model takes 300 ms; the same request to a cold model takes nine seconds because the weights must be loaded from disk first. The variance in service time is enormous and bimodal.
Two-lane priority: interactive jumps batch
The minimal design that solves the problem has two lanes: - Lane 0 (interactive): requests from user-facing surfaces. Drained first. - Lane 1 (batch): requests from background jobs - eval harnesses, post-mortem generation, cron pipelines. Drained only when lane 0 is empty.
The watchdog: detecting wedged runners
Even with priority queues, the inference engine itself can go bad. A model load that hangs, a network filesystem stall, a runner process that has crashed but whose parent has not noticed - any of these can leave a request in-flight forever. From the proxy's perspective, the engine is still "busy with the current request" and refuses to dequeue another. The fix is a watchdog: a small thread that checks, periodically, how long the current in-flight request has been in flight. If it exceeds a threshold (a few minutes, calibrated against the longest expected legitimate response), the watchdog flips a flag - "runner stuck" - that gets surfaced via a health endpoint so the operator can read it and know it is time to intervene. A more aggressive variant gives the watchdog permission to kill the stuck runner process itself and let it respawn - trading a little robustness against the risk of killing a legitimately-slow request, worth it for systems where operator response time is measured in hours, not minutes.
Admission bounds and graceful 503
Every cap in the system - interactive queue depth, batch queue depth, per-request timeout, watchdog wedge threshold - exists to prevent the system from accepting work it cannot complete. When a cap is exceeded, the proxy must respond with a deterministic, structured error that the caller can interpret. The minimum-honest 503 response includes three things: the reason (a short machine-parseable code like "interactive_lane_full", "upstream_unreachable", or "runner_stuck"), a retry hint (seconds to wait, or zero if the situation isn't expected to clear soon), and a queue depth snapshot, so the caller can back off intelligently or fall back to a smaller model.
Worked picture
At time 0, a scheduled batch job submits a 40-second post-mortem generation request. The engine begins processing it. At time 5 s, an interactive user opens a chat and submits a question. The proxy enqueues the request in lane 0. At time 45 s, the batch completes. The proxy pops the interactive request next, even though it arrived later than the batch. The user gets a two-second response and sees a 42-second wait - which is the right behaviour, because the platform could not preempt the in-flight batch. Compare against FIFO: the interactive user would still wait 42 seconds, but any subsequent interactive request that arrived at time 46 would also wait behind whatever batch jobs had queued in the meantime. Under priority, only the in-flight batch produces a wait; subsequent interactives are always served first. The difference is invisible at low load. It is the difference between a platform that feels responsive and one that feels broken under cron storms.
What breaks in production
A common failure mode after introducing a priority gateway: the gateway issues a 503, the caller retries immediately, the gateway issues another 503, ten retries happen in one second, the caller logs "ten failures, alerting operator," and the operator wakes up to a false alarm at three a.m. The fix is the retry discipline from Chapter 47: exponential backoff with jitter, on the caller side. The gateway can suggest a retry interval via a response header, but the caller is the one that must respect it. A well-behaved caller and a well-bounded gateway are both parts of the same system; either one missing makes the other ineffective.
Key idea
A priority admission controller is the difference between a platform that degrades gracefully under load and one that simply stops working. The design is small - a priority queue, two lanes, a watchdog, deterministic 503 responses with reasons - but the operator and user experience under contention is transformative. Build the gateway before you need it; once you need it, you will be too busy fixing the consequences of not having one to find the time to build it.