Serving Models

Serve requests, not merely forward passes

A model can fit on a GPU and still make a poor service. Requests arrive at different times. Histories occupy different amounts of memory. Long prefills can interrupt ongoing generation. A useful deployment meets latency constraints while keeping enough work available to use the hardware.

Start with the workload: interactive chat, offline generation, agents, edge inference or real-time audio. These workloads do not ask for the same optimum. More total tokens per second can coexist with slower individual responses.

Separate model work, resident state and service scheduling. Follow those constraints from one request to the whole deployment. The interactive examples below are original teaching models, not production benchmarks.

Follow a request through the service

client → gateway/authentication/rate limit → validation
       → standard or priority admission queue
       → application/model router → inference engine
       → output policy gate → stream to client
registry/configuration and telemetry accompany this path

The gateway authenticates callers and enforces quotas. A token bucket permits a bounded burst while replenishing allowance at a declared rate. One token per request is simple, but ignores differences in prompt and output cost. Token- or compute-based accounting needs an explicit policy.

Validation checks schema, size, parameters and model compatibility. Normalize only what the application permits. Assign priority before admission. Separate queues do not prevent starvation: track queue age and reserve service when fairness requires it.

Application orchestration can select a faster model, a stronger model or a multi-turn workflow. Infrastructure orchestration manages workers and resources. Kubernetes does not automatically infer task difficulty or choose a reasoning policy. Model switching also requires compatible formatting and state reconstruction.

Version models, tokenizers, templates and policies through a registry. The original sketch proposes API gateways, Redis, Nginx, FastAPI/Pydantic, Kubernetes and observability tools. These are possible components, not mandatory choices or evidence of million-user capacity.

A model registry can also support tuning and experiment tracking. Auth0, Weights & Biases and Grafana are examples from the high-level design, not required service dependencies. After policy approval, the application can return the stream through the coordinator and gateway without repeating input validation. Cancellation and backpressure still need an explicit return-path contract.

Prefill and decode expose different work

Prefill processes prompt positions in parallel under the causal mask. It builds cache state and produces logits at the final prompt position. Sampling those logits can produce the first output token. Later decode calls consume generated tokens and predict following tokens.

These are tendencies. Short batch-one prefills can underutilize hardware. Large decode batches can make linear layers compute-bound. Long-context decode can remain cache-bandwidth-bound. Measure actual shapes rather than treating the phase label as a bottleneck diagnosis.

First token, cadence and goodput

A service-level objective (SLO) states an acceptable outcome. Report raw throughput and SLO attainment together. Mean TPOT can hide occasional gaps that make streaming unpleasant. More requests per second does not necessarily mean more output tokens per second.

Record three scheduler diagnostics alongside these outcomes. Decode occupancy compares occupied slots with configured capacity. Prefix reuse compares cached prompt tokens with eligible prompt tokens. Transfer overlap measures how much handoff is hidden behind prefill. Define each measurement interval and denominator; none alone establishes goodput.

What the cache preserves

A key–value (KV) cache saves earlier attention projections. A new token forms Q, K and V, appends state and reads permitted history. Previous positions do not need to pass through all projections again.

For a fixed causal model, caching avoids repeated prefix processing. Feed-forward work becomes linear in generated positions. Dense attention still reads a growing history, so accumulated attended work and traffic remain quadratic in the simplest full-attention accounting.

Distinguish logical context length, physical allocation and positions actually attended. A runtime may retain blocks that a local layer no longer needs. An advertised context limit does not prove efficient eviction or retrieval.

Count KV heads and distinct stored tensors

This assumes equal lengths and uniform, distinct per-layer caches. For ragged histories, replace B·S with the sum of lengths. Cap local layers at the window actually retained. For shared or hybrid state, sum distinct stored representations instead of multiplying all layers by one constant.

Cache dtype can differ from weight and activation dtype. Quantization scales, alignment, page tables and temporary workspaces add memory beyond the payload formula.

For one stored token in one sequence, the same distinct K/V and uniform-layer assumptions give $M_{\mathrm{KV/token}}=2LH_{\mathrm{kv}}d_{\mathrm{head}}q$. This unit cost is useful when reserving room for an output history to grow.

The 1 GiB example

Llama 3 8B has 32 layers, 32 query heads, eight KV heads and head width 128. A BF16 cache at 8,192 positions occupies exactly one GiB per sequence. Using query heads incorrectly predicts four GiB.

This is 128 KiB of payload per token per sequence. Multiply by concurrent histories, not only newly generated tokens. Output generation grows the cache even when the initial prompt is unchanged.

A cache can fit and still be slow

Batching shares weight loads across requests. Requests usually have different histories, so the engine still reads each attended cache. A larger batch can amortize weights while increasing KV traffic almost proportionally.

Cross-layer sharing saves stored capacity but may leave repeated reads. Faster-cache residency or kernel fusion must realize traffic reuse. Sparse retrieval also pays index-selection cost. Fewer selected values do not make the indexer free.

Change the cache at the architectural level

Architectural compression changes preserved information. Runtime paging changes where unchanged logical histories reside. Keep these axes separate when comparing quality, storage and latency. Attention owns operators and historical branches; Precision and Quantization owns numerical tradeoffs.

At fixed layers, head width, positions and dtype, the MHA-to-GQA payload ratio is $\frac{M_{\mathrm{MHA}}}{M_{\mathrm{GQA}}}=\frac{H_q}{H_{\mathrm{kv}}}$. MQA sets the KV-head count to one. This comparison concerns distinct stored tensors, not a guaranteed latency ratio.

Logical histories and physical pages

Contiguous allocation up to maximum context wastes capacity for short histories and complicates growth. Paging divides storage into blocks. Each request maps logical blocks to physical blocks through a table, allowing independent placement.

Ragged kernels use each request’s actual length instead of reading batch-wide padding. Paging reduces allocation fragmentation; ragged execution reduces unnecessary work. They solve different parts of the problem.

Sharing needs reference counts and safe lifetimes. Mutable suffixes require private storage or copy-on-write. Free a page only after all consumers, transfers and in-flight kernels finish. Speculative branches make these ownership rules especially important.

Reuse identical causal prefixes

Identical causal prefixes have identical KVs only under the same model and compatible execution contract. Token IDs, positions, adapters, multimodal inputs and cache format belong to the identity. Matching visible text is insufficient.

A trie or radix tree records shared prefixes. LRU-like eviction can release cold branches while retaining useful ancestors. Measure reused prompt tokens and saved work, not only binary request hits. A short system-prompt hit and a huge history hit have different value.

State belongs to the checkpoint that created it. Resuming RL trajectories after a weight update may deliberately reuse older caches, but does not reproduce a fresh forward pass under the newer checkpoint. Retain policy-version provenance.

Cache hits constrain placement

Affinity routing improves hits by sending related requests to the worker owning their histories. It can overload that worker while others sit idle. Balance saved prefill work against queue delay, transfer and fairness.

CPU memory or SSD can retain cold histories. Reloading helps only when transfer and synchronization are preferable to recomputation. Global movement can erase the gain when caches are large and future reuse is unlikely.

GLM-5 reports rollout-ID-aware consistent hashing alongside FP8 inference, MTP and prefill/decode separation. Kimi K3 keeps active state on GPU and evicted idle prefixes in an external CPU pool. These are designs, not proof that affinity or offload always improves goodput.

GLM-5 report ↗

Admit work when a slot becomes free

Static batching waits for a group to finish before admitting the next group. Unequal output lengths leave empty slots. Continuous batching replaces finished requests at step boundaries while other requests continue.

Admission must constrain active slots and memory. Reserve room for histories to grow; a batch that fits now can fail later. Cancellation releases state only after in-flight work finishes.

Configured capacity does not create traffic. Waiting to fill a batch raises queue latency. Under light interactive load, a partially filled batch can be the correct choice. Offline generation may allow much larger latency.

Bound the interruption, not just the batch

A long prompt can delay decode requests sharing the engine. Chunked prefill divides prompt work into bounded pieces and interleaves them with decode steps. The scheduler chooses an iteration-level token budget.

Smaller chunks protect cadence but can increase prompt completion time and scheduling overhead. Larger chunks improve prefill efficiency but expose longer interruptions. Neither removes resource competition.

Track prefill tokens admitted per iteration, oldest queue age, decode gaps and cache growth together. Finishing prefill still leaves decode admission and memory requirements. One batch-size setting cannot describe the whole schedule.

Weight reuse and KV traffic are different

The model separates KV traffic from linear work. Arithmetic and weight loading can overlap within the linear stage. Attention arithmetic, cache writes, collectives, sampling, launches and scheduling are omitted. The expression is not a universal sum of independent hardware lower bounds.

For MoE, batch weight traffic depends on the union of selected experts. Active parameters per token do not determine that union. Use traces or an expert-reuse model before applying a dense estimate.

One-token decode throughput is approximately B divided by step time. Speculation changes accepted output per verification and adds draft work. Compute, Memory, and Performance explains achieved rates and accounting boundaries.

Derive the precision-dependent crossover

BF16 weight bytes $q_w=2$ give $\frac CW$. Halving weight storage at unchanged arithmetic rate halves the crossover. Doubling arithmetic rate doubles it. Use the rate of the actual kernel, not another precision’s advertised throughput.

A weight-to-activation-width shorthand can hide a normalization convention. Activation bytes do not independently enter this displayed equality. Retain the byte/FLOP derivation when comparing precision.

The transition concerns linear work only. KV bandwidth, capacity, queueing and tail constraints can make another batch preferable. It is not a universal economic optimum or a fixed “300 times sparsity” rule.

Budget distributions and headroom

KV_budget = HBM - weights - workspaces - graphs - reserve
KV_demand = sum(cache_bytes(history) for history in active)
admit only if demand + growth reserve <= KV_budget
then check queue and decode latency

Use prompt and output distributions. A few long contexts can dominate occupancy and tails. Prefix reuse saves work but still occupies shared storage. Beam search and speculation create temporary state.

Dividing net memory by equal-length cache payload is only a first estimate. Add paging, fragmentation, retained idle histories and growth. Check that local-window blocks are actually released.

Capacity is feasibility, not throughput. A larger group can provide enough memory for a useful batch and enough bandwidth for faster decode. The smallest group fitting weights can require more replicas or miss cadence.

Keep large state stationary

Keep large weights and histories stationary when possible; move smaller activations. Training-style just-in-time weight gathering can be expensive during serial decode because network bandwidth is usually far below aggregate HBM bandwidth.

Tensor parallelism keeps permanent weight shards. Pipeline parallelism increases capacity but needs concurrent work to fill stages. Expert parallelism adds dispatch and routing costs. Smaller local weight storage can coexist with more exposed communication.

Shard KV heads when enough heads exist. Batch or sequence sharding extends capacity when heads are few. Sequence sharding needs distributed attention reductions; layout changes can add AllToAll. Avoid replicating huge caches merely because weights fit.

Parallelism owns layouts; GPU Communication owns startup, transport visibility and reuse. Validate full decode steps after changing either.

Colocation, interleaving and disaggregation

Disaggregate when measured isolation or specialization gains exceed handoff costs. Short prompts, low load, weak networks or operational simplicity can favor colocation. Neither design universally replaces the other.

Separate pools can use different hardware or parallelism degrees and can retain more prefix history on the prefill side. That benefit depends on actual free memory after weights and workspaces. Steadier decode occupancy is a possible scheduling result, not guaranteed by separation.

Transfer cache before decode can use it

Transfer completed layers while later layers compute to hide some handoff. Count exposed time, not every transfer duration. Direct transfers, topology-aware placement, compressed caches and insertion headroom can help.

Destination shards and formats must be compatible. Define ownership, visibility, cancellation and retries. A failed worker requires recomputation or restoration of a surviving cache. Free bytes alone do not make insertion safe.

Match request rates across pools

This saturation estimate is not a queueing guarantee. Lengths, hits and occupancy vary. Transfer can cap throughput below both pools. Leave burst and failure headroom.

An assigned example uses two prefill workers at two seconds per request: one request/s. One decode worker with batch ten, 50 ms steps and 200 output tokens also supplies one request/s. Doubling only prefill leaves decode as the limiter. These are illustrative times, not measurements.

Measure saved prefill tokens and actual decode occupancy before changing ratios. A high request-level hit rate can still leave most prompt work intact.

Encoding can be a third pool

Vision or audio encoding can precede language-model prefill. Encode–Prefill–Decode (EPD) adds a third independently scaled pool. Encoder features must reach the language pool before relevant prompt work starts.

GLM-5.3-Flash reports EPD, intra-node tensor parallelism for KDA and the LM head, ReplaySSM, W8A8, mixed INT8/FP8/BF16 caches and Layer Split. The provider describes deployment across tens of thousands of Chinese accelerators.

Its reported 3× end-to-end improvement compares against its initial implementation on the same hardware, not other models. Claimed 3.0× attention-compute and 4.4× BF16-cache reductions versus GLM-5.3 are analytical architecture comparisons without a complete reproducible formula table.

IndexPool scores four-token pools, then expands selected pools to raw KV positions. Generic implementations can retain raw keys and gates; optimized storage realizes the claimed memory reduction. Fewer index candidates do not imply fourfold less main attention work.

GLM-5.3-Flash release ↗

Skip dependencies only when the architecture permits it

DeepSeek-V4.1-Flash changes prefill dependencies. Twenty lower causal encoder blocks produce representations for global memory in twenty upper decoder blocks. Most prompt positions can skip upper-block execution.

Generated tokens still use all forty blocks. Reported active backbone parameters are about 8B/prefill token and 16B/decode token. Local decoder caches require replay, which is approximate because stacked local layers can depend beyond one window. Post-training simulates that approximation; edge cases remain uncharacterized.

The reported global cache is 890 bytes/input token: 890 MB decimal, about 0.83 GiB, at one million tokens. Local state, weights and workspaces are additional. FP4 global values are dequantized before attention; local cache stays FP8. Storage compression is not native FP4 attention arithmetic.

Engram tables use host prefetch through background RDMA in serving, but stay in GPU memory in the described RL rollout setup. Reference inference code does not automatically reproduce production CED skipping, replay or DSpark scheduling. Check the actual deployed path.

DeepSeek-V4.1 report ↗

Compose storage reductions without conflating them

Gemma 4 composes local/global attention, partial RoPE and shared representations. E2B uses a 4:1 local/global ratio; others use 5:1. Global layers of 12B, 26B-A4B and 31B set $V=K$. With rotary fraction $p=0.25$, storage changes from approximately $2d$ to $d+pd=1.25d$: 37.5% less global cache, not total model memory.

Edge-model KV-sharing ratios are $\frac{20}{35}$ and $\frac{18}{42}$. Shared storage need not remove repeated reads. Weight, active expert weight and KV footprints are separate.

At 32k text context, reported INT8 KV adds 0.05 GB for E2B through 1.10 GB for 31B. Vision W8A8 forward memory falls from 400 to 200 MB. Reported 44% lower vision latency than Gemma 3n uses newer on-device hardware, not a hardware-matched quantization ablation. Audio disk footprint falls from 390 to 87 MB with mixed quantization. Block scalar scales support stable FP16 inference.

Gemma cache representation counts the retained key/value components explicitly.

Gemma 4 report ↗

Manage growing and recurrent state together

Kimi K3 manages fixed-size KDA matrix/convolution state and growing MLA KV together. GLM-5.3-Flash has eleven of forty-five growing-cache blocks, about 24.4%; the rest retain recurrent and short-convolution state. Fixed state compresses history rather than preserving arbitrary token-level retrieval.

A rejected draft suffix cannot be removed from a recurrent matrix by slicing its token dimension. K3 saves projected draft inputs and replays the accepted prefix instead of copying every intermediate state. Recurrence, attention cache and committed tokens must agree.

K3 uses MXFP4 routed weights, MXFP8 expert activations and higher precision for sensitive components. QAT continues through SFT and RL. Fixed state, growing state and numerical formats need separate memory accounting.

Kimi K3 report ↗

Shared weights do not imply shared cache

Nanbeige4.2 executes one physical stack twice. Both passes share weights, but its selected full-loop configuration keeps distinct caches. Sharing caches halves KV storage but produced smaller quality gains.

The authors report about 75% of standard-model token-processing efficiency for two passes, with marginal quality gains from deeper recurrence. The full loop-depth table and serving-cost breakdown are undisclosed. Shared weights do not remove sequential computation or effective-depth state.

Recurrent and Looped Models owns the equations and wider evidence. Budget executed state instances, not physical parameter count alone.

Nanbeige4.2 report ↗

Verification also has a state cost

Drafting changes committed tokens per target verification and adds proposal work, candidate state and rollback/replay. The engine must retain accepted state and discard rejected suffixes safely.

Gemma 4’s four-layer autoregressive drafter cross-attends to the target cache, avoiding separate drafter prefill. Edge-model token clusters reduce the final projection from d×262,000 to d×4,096 while retaining similar reported acceptance. Proposal latency matters, not only prediction accuracy.

GLM-5.2 shares selected indices over four-layer groups. Its reported 2.9× lower per-token FLOPs at one-million context is not throughput or proportional KV reduction. A seven-step MTP ablation progresses from acceptance length 4.56 to 5.10 with IndexShare/KVShare, 5.29 with rejection sampling and 5.47 with total-variation loss. Cumulative changes do not isolate each contribution.

Speculative Decoding owns acceptance, pending state, drafter families and DSpark. Relate acceptance to measured proposal/verification time and memory.

GLM-5.2 release ↗

A guard needs its own failure policy

Policy gates add latency, compute and failure paths. Shieldstral-like classifiers answer a conditional policy question with one token. Short output is logically cheaper than long reasoning, but the paper reports no production latency, throughput, memory or cost comparison.

A cascade can allow clear safe cases, block violations and escalate uncertainty. Restricted yes/no normalization does not prove calibration. Measure false positives and false negatives by language, domain and severity.

Version policies, protect trusted inputs, retain audit/appeal records, monitor drift and red-team injection. Define fail-open or fail-closed behavior by risk and deterministic fallback when the guard is unavailable. A classifier is not a complete enforcement mechanism.

Structured Decision and Safety Models owns conditional scoring, calibration and cascades.

Shieldstral report ↗

Compare cost at an acceptable latency

Compare full deployments: replicas, CPUs, memory tiers, network, headroom, idle time and failures. A larger replica can cost less per useful token if it admits a useful batch or reduces required replica count.

Output length changes cost and request throughput. Disaggregation duplicates weights and adds transfer but can improve goodput. Prefix reuse saves prefill, not output generation. Public API prices do not establish a universal five-to-one input/output cost ratio.

For each topology, calculate net cache capacity, estimate achieved rates and collectives, measure or simulate the workload distribution, enforce SLOs, then account for realistic fleet occupancy. Devices per replica alone is not an economic comparison.

Real-time audio has another constraint

RTF below one means faster-than-real-time in the measured setting. Equal to one leaves no jitter margin. Average RTF does not establish fast first audio or uninterrupted chunk delivery.

N/RTF assumes divisible or efficiently scheduled work and approximately linear demand. Isolated RTF does not prove concurrent scaling. A slow individual stream does not become real-time merely because fleet aggregate throughput is sufficient.

Subtract shared weights and workspaces from memory first. KV, decoder state and buffers depend on the model. Include codec/protocol overhead and admission headroom. Streaming requires chunk deadlines and jitter reserve, not just three aggregate counts.

Read the 20-stream example as an ideal budget

The declared minimum is twenty streams, so compute limits this ideal example. A nominal 24 GB device does not provide all 24,000 MB for stream state after model allocation. Interpret it as a hypothetical net budget and replace it with measured free memory.

For compute, evaluate smaller/distilled models, supported precision or faster kernels. For memory, share weights, compress compatible KV or offload inactive state. For network, choose an appropriate codec, bitrate or sample rate. Opus and learned codecs differ in compatibility, quality and processing cost.

Remeasure quality and concurrent deadlines after each change. Faster kernels help only the actual model path. Additional GPUs help when work partitions or schedules efficiently; they do not guarantee every stream’s cadence.

Autoregressive systems such as Bark can retain decoder KV state as well as output buffers. TensorRT, FP16/INT8 and FlashAttention are possible implementation choices only when the model and kernels support them; none guarantees this stream count. For audio transport, Opus and EnCodec have different codec contracts. Lowering a sample rate, for example from 24 kHz to 16 kHz, can reduce demand but changes the quality and compatibility assessment.

Observe the limiter before changing the layout

1. Record request, prompt, output and modality-size distributions; include cache-hit tokens.
2. Trace queues, prefill, exposed transfer, decode gaps and guard latency separately.
3. Budget resident state, growth, workspace, pages and idle histories.
4. Inspect achieved HBM, arithmetic and collective cost on the critical path.
5. Change one admission, chunking, placement or sharding choice at a time.
6. Compare goodput, tail latency, quality and recovery together.

The original service sketch requests sessions, multiple models, near-real-time delivery and at least 99.9% availability. These are requirements, not demonstrated outcomes. Registries and monitoring support them only when recovery, backpressure, versioning and state ownership are implemented.

Sources and connected topics

• JAX Scaling Book: Inference ↗; Applied Inference ↗; Roofline ↗.
• Llama 3 model card ↗; PagedAttention ↗; DistServe ↗; Splitwise ↗.
• Original high-level design reference ↗. Request path reconstructed as a conceptual diagram, not an endorsed vendor blueprint.

Report-specific observations retain their original provenance. Full reports and references belong to Frontier Model Training Recipes, Small and On-Device Model Recipes and Decision and Safety Models. Use Profiling to find the exposed limiter.