Design an LLM Query Batching System
You're asked to design a batching system for LLM inference. One GPU can process up to 100 inputs at once, requests arrive continuously, and users wait synchronously for responses.
Start by asking whether to treat the model as a black box. If so, we can focus on the API, queue, batching policy, worker pool, and failure handling rather than inference internals. Confirm that boundary early because the two scopes lead to different designs.
Clarify the requirements
Clarify two questions before we design the batcher:
- Is the caller waiting? Synchronous callers mean latency targets and an async-to-sync bridge at the end. An offline scoring job means large batches and throughput as the only metric. Assume synchronous for this breakdown.
- What are we optimizing? "Maximize utilization" and "minimize latency" pull in opposite directions. Ask which one wins, and what the acceptable ceiling is on the other.
Then state the black-box assumption: the model takes up to N inputs and returns N outputs in roughly fixed time. We can now design the system around it.
The core tension: batching versus latency
Processing one request at a time wastes most of the hardware, so requests get grouped into a batch. But a batch has to fill before it runs, and every millisecond spent waiting for it to fill is latency the caller pays.
That tradeoff determines how long we wait, when we flush a batch, how we handle overload, and how an asynchronous pipeline returns a result to a caller waiting on HTTP.
High-level architecture
- API layer. Accepts the request, assigns it an ID, and holds the caller's connection open while the work happens elsewhere.
- Request queue. Decouples arrival rate from processing rate and absorbs bursts.
- Batcher. Groups queued requests and decides when a batch is ready to run.
- Scheduler. Picks which worker gets the next batch, based on which one has capacity.
- Workers. Each owns a GPU, runs the batch, and emits results.
- Result path. Carries each output back to the request that's still waiting for it.
Deep dive 1: when do you flush a batch?
Let's flush a batch when either of two conditions is met:
- Size threshold. The batch reaches the hardware's maximum, 100 inputs in the stated prompt. Under heavy load this is what fires, and utilization is at its best.
- Time threshold. A maximum wait, say 20 milliseconds, measured from when the first request in the batch arrived. Under light load this is what fires, so a lone request at 3am doesn't wait forever for 99 friends.
The time threshold is our latency dial, and it should be tunable rather than fixed. We can also make the wait adaptive, shrinking it when the queue is deep (batches fill on their own) and stretching it when traffic is thin (waiting a little buys real utilization).
Deep dive 2: returning an async result to a synchronous caller
Candidates consistently report this as the follow-up that separates answers: if you queue requests asynchronously, how do you return the response to the same user synchronously?
The caller made one blocking HTTP request. Internally, that request became a queue entry, got grouped with strangers, ran on a worker, and produced an output somewhere else entirely. Something has to reunite the two.
- Correlation ID. Every request gets an ID that travels with it into the batch and comes back attached to its output. This is the non-negotiable part.
- The waiting mechanism. The API instance that holds the connection subscribes to results for its outstanding IDs, typically over a pub/sub channel or a result queue it consumes. When the answer arrives, it writes the HTTP response.
- Which instance is waiting. With multiple API servers, the result has to reach the specific instance holding that connection. Either publish results to a topic every instance subscribes to and let each pick out its own IDs, or route by instance ID recorded when the request was accepted.
- Timeouts. The caller won't wait forever. Define what happens when the deadline passes: return a 504, and make sure the in-flight work is either cancelled or harmless to discard.
- Streaming as the alternative. If the product can accept incremental output, the answer changes shape: return a stream immediately and push tokens as they're produced, which sidesteps the blocking problem.
Deep dive 3: choosing which GPU gets the next batch
With a pool of workers rather than one, batch assignment becomes a scheduling problem we need to solve explicitly.
Round-robin ignores the fact that batches take different amounts of time and workers drift out of sync. Use capacity-aware routing instead:
- Workers report their state by heartbeat to the scheduler: busy or free, current queue depth, and estimated time remaining.
- The scheduler assigns each batch to the worker with real capacity, effectively least-outstanding-work rather than least-connections.
- Treat stale state as failure. A worker that stops heartbeating gets removed from the pool, and its in-flight batch is reassigned.
One candidate described proposing a GPU-aware load-balancing layer that tracks availability and routes work accordingly. Use a scheduler with a live view of the fleet rather than a fixed round-robin balancer.
Deep dive 4: failures, backpressure, and scale
- Queue overflow. When arrivals outpace the fleet, the queue grows without limit and every caller times out. Cap the queue length and shed load at the API layer with a 429 once it's full, so callers fail fast instead of waiting for a response that will never come in time.
- Worker crash mid-batch. The whole batch is lost, not one request. Either re-enqueue every request in it, which requires each to be idempotent or simply re-runnable, or fail them individually. Say which you're choosing and why.
- Poison inputs. One malformed input that crashes the worker takes 99 healthy requests down with it. Retry a failed batch by splitting it, so a bad input is isolated rather than repeatedly killing full batches.
- Fairness across tenants. One caller submitting ten thousand requests must not fill every batch. Per-tenant concurrency caps and weighted selection when forming batches keep one heavy user from monopolizing the fleet.
- Autoscaling signal. Scale on the age of the oldest queued request rather than raw queue depth, since that's what maps to the latency a caller actually experiences.
Deep dive: model internals
If the model internals are in scope, move into the ML-specific constraints below.
Generation isn't one forward pass. A classifier does one pass and returns. An LLM produces one token per step, each depending on all the ones before it. Processing the prompt (prefill) is compute-bound and parallelizes well; generating tokens (decode) is memory-bandwidth-bound and sequential. That's why time-to-first-token and tokens-per-second are reported separately.
Static batching wastes the chip. Requests in a batch finish at wildly different times, one generating 10 tokens and another 2,000, so the whole batch holds the GPU until its longest member finishes. Continuous batching admits and retires requests at every decode step, keeping the batch full. It's the biggest throughput win available in LLM serving.
Memory, not compute, caps concurrency. Each in-flight request holds its attention state, the KV cache, in GPU memory for the whole generation, and long contexts hold gigabytes. That reframes "up to 100 inputs per batch" as a memory budget. Paged KV allocation avoids fragmentation, and prefix caching shares the cache for a common system prompt across requests.
The cost math. A GPU at roughly $2/hour producing ~1,000 tokens/sec yields about 3.6M tokens/hour, a floor near $0.55 per million tokens before utilization losses. A fleet at 30% utilization triples the real cost per token, which is what makes batching an economic decision rather than a performance tweak.
Common pitfalls
- Designing the model instead of the system. If the prompt scopes the model as a black box, minutes spent on inference internals are minutes lost. Confirm the boundary, then design the surrounding system.
- No answer for async-to-sync. Queueing requests without explaining how the response reaches the waiting caller leaves the design incomplete.
- Round-robin scheduling. Ignores that workers have different amounts of work in flight.
- An unbounded queue. Turns an overload into universal timeouts instead of fast, honest failures.
- Ignoring partial failure. A batch is a unit of work whose failure affects many independent callers.
Leveling signals
Related lessons
Serve a multi-turn chat product backed by a large language model.
Answer customer questions from company data and take actions on their accounts.