Speculative Decoding

Turn guesses into parallel work

At a small decoding batch, weight and KV traffic can dominate arithmetic. A target step has little work per byte transferred. Scoring a known block resembles a small prefill: positions can run together under a causal mask.

The hardware advantage is conditional. A saturated engine may have no spare verification capacity. A large draft tree can improve single-user latency while reducing aggregate throughput. Measure both.

One round and shifted logits

1. The drafter proposes γ tokens and records their actual proposal probabilities.
2. The target scores that tentative continuation, obtaining γ proposal distributions and one distribution after the last proposal.
3. Verification checks proposals from left to right.
4. At the first rejection, commit a corrected token and discard every later proposal.
5. If every proposal survives, sample a bonus target token.

A rejected proposal changes the prefix. Every later target distribution was computed for a history that did not occur. A plausible later token is therefore not reusable just because it looks correct.

The target must have the first distribution available from the prefix or pending last-token input. Feeding only draft tokens and comparing unshifted logits creates an off-by-one error.

Exact sampling: account for every probability

At one position, sample x from the drafter q. The acceptance probability and accepted mass are:

The missing probability mass is exactly the positive difference p−q. Mixing the accepted branch with the rejection branch restores every target probability.

α = 0.8. Rejection occurs with probability 0.2. The residual chooses dog or bear with probability 0.5 each. Sampling directly from p after rejection would double-count already accepted mass.

A token with q(x)=0 cannot be proposed through q, but the residual can still produce it if p(x)>0. A proposed token has q(x)>0 under exact arithmetic. Handle underflow through stable log probabilities rather than silently defining a ratio for an impossible proposal.

Greedy, sampling and the real sampler

For greedy decoding, accept a proposal while it equals the target argmax. At the first mismatch, use the target argmax. This reproduces the deterministic target sequence if tie-breaking and numerical behavior agree.

Stochastic decoding requires the acceptance/residual rule. Greedy equality is not evidence of stochastic exactness. The probabilities must include the transformations that define the actual target and draft samplers: temperature, truncation, token bans and repetition penalties.

Target and drafter need not use identical distributions or identical sampler settings. The ratio must use the actual q that produced the proposal and the actual p whose outputs we want. A stored raw softmax is wrong if the draft was sampled from a truncated distribution.

The simple token-wise implementation assumes compatible token IDs and vocabularies. Vocabulary translation is possible, but it needs correct probability accounting at retokenization boundaries. It is not a safe first optimization.

Decoding and Sampling → separates distribution transformations from acceleration.

Accepted length is not speed

Let cᵢ be the acceptance probability at position i conditional on earlier positions surviving. Multiplying these conditional probabilities gives the survival probability aⱼ through position j.

The ideal speedup assumes that verifying the entire block costs one ordinary target step. Serial drafting adds γ times the draft-step cost. At perfect acceptance, all γ proposals survive and the round also commits its bonus token.

This simplified model explains diminishing returns. Extra positions have smaller survival probability but still consume draft and verification resources. Real verification time is not constant, and acceptance varies with context and position. Add sampling, synchronization, cache and scheduler time to the measurement.

Chains, trees and visibility

A chain commits only one proposed path. A tree offers alternatives at ambiguous positions. Flatten its nodes for verification, but give each node access only to the shared accepted prefix, its ancestors and itself.

fox
├── jumps
│   ├── over
│   └── high
└── runs
    └── away

The node “over” may attend to fox and jumps. It must not attend to runs or away. Position IDs follow depth rather than flattened storage order. Parent arrays and an ancestor mask make this constraint explicit.

An ordinary triangular mask on the flattened nodes leaks sibling information. Tree verification also needs a method-specific proposal and sampling rule. A correct mask alone does not prove lossless stochastic tree sampling.

After choosing a valid path, compact its KV entries into the next logical prefix. Branch states cannot remain addressable as if they belonged to that path. Compare extra accepted length with the extra target nodes and irregular-kernel cost.

How the bottleneck moved

The separate-model baseline remains useful. Prompt or n-gram lookup can be almost free when continuations repeat. New architectures do not erase those workload-specific advantages.

Choose a drafter by its dependencies

Choose the simplest family that resolves the measured bottleneck. Prompt lookup can beat a learned drafter on repeated continuations. Target-attached heads save a second backbone but can lose suffix coherence. A parallel backbone saves draft depth, while its verifier still pays for every admitted position.

Medusa and native MTP

Medusa attaches several heads to the target’s final hidden state. Head j predicts a future offset. Candidate combinations form a sparse tree. Reusing the target avoids a separate draft backbone and prefill.

Independent heads do not observe the concrete token selected at earlier offsets. Several plausible marginals can form an implausible joint path. Medusa-1 freezes the target and trains the heads. Relaxed typical-acceptance variants must not be called exactly distribution-preserving.

DeepSeek-V3’s sequential MTP modules combine a preceding depth representation with a preceding future-token embedding. DeepSeek-V3 reports one extra inference depth, 85–90% extra-token acceptance and 1.8× tokens/s in its setup. This does not predict throughput for another engine or a longer unroll.

EAGLE: reuse target computation

EAGLE predicts the target’s second-to-top-layer feature autoregressively, then uses the target LM head for draft logits. The sampled token conditions the next feature prediction. It resolves which discrete continuation the feature should follow.

EAGLE-2 spends a tree budget using context-dependent path confidence instead of a static tree. Promising paths receive expansion. Better confidence allocation still has to justify extra verification nodes.

EAGLE-3 removes direct feature regression, predicts tokens, and fuses features from several target layers. Its “training-time test” exposes later draft steps to earlier model-generated states. This reduces the mismatch between teacher-forced training and recursive inference.

These methods preserve a sequential draft dependency. Their quality advantage competes with latency that grows over draft depth.

DFlash and parallel block drafting

DFlash processes an anchor and masked future slots in one bidirectional draft pass. It concatenates selected target-layer features, projects them into draft width, applies RMS normalization, and injects them as key/value context in every draft layer. It reuses the frozen target embedding and output head.

The target features must stop at information available to the anchor. Clean future target states would leak the answer during training. Separate packed blocks must not attend to each other.

All slots run together, so a deeper drafter need not run once per future token. Slots can exchange latent information, but they do not observe the concrete sampled predecessor token. This remaining dependency gap causes suffix decay.

The target is frozen for the basic training setup. Random response anchors form masked blocks, supervised by target sequences or distributions. Position-decayed weights emphasize early positions, whose failure invalidates the suffix.

DFlash2: local mixing and a candidate walk

DFlash2 uses two-tap grouped dynamic causal convolutions around attention and MLP sublayers. A low-rank predecessor-conditioned selector walks top-K unary candidates.

The source’s September 2026 status records code/blog evidence rather than a peer-reviewed DFlash2 paper. Treat that as a dated provenance statement, not a fresh claim about every current release. The training recipe and exact proposal accounting still need verification for a reproduction.

DSpark: parallel capacity, cheap dependencies

DSpark addresses two failures of independent block proposals: incoherent suffixes and wasted verification capacity. A DFlash-style backbone emits states hₖ and unary logits Uₖ for all slots. A light sequential head adjusts those logits without rerunning the backbone.

A parallel proposal might combine “of problem” or “no course.” Conditioning the next slot on the preceding sampled token can favor “of course” and “no problem.” This is a synthetic explanation, not a reported language-quality result.

An RNN head can carry more of the within-block prefix. The recorded experiments found modest additional long-block gains, with more deployment complexity. The Markov head is the default described in the source.

DSpark counts the anchor input position as the first proposal prediction. One anchor plus γ−1 masks can produce γ proposals. Do not import another drafter’s offset convention unchanged.

Confidence and verification admission

A confidence head estimates conditional acceptance cₖ using hₖ and the preceding token’s Markov embedding. Its soft training label is the target–draft distribution overlap, 1−TV(pₖ,qₖ), not whether one already sampled token happened to pass.

Cumulative survival controls expected return. Products amplify calibration errors. DSpark calibrates one position at a time on held-out data. It minimizes cumulative-product calibration error while keeping earlier calibrated positions fixed. Ranking quality alone is insufficient for an expected-throughput calculation.

Under the source’s simple capacity model, the scheduler extends prefixes in survival-probability order while estimated throughput improves. A later token is useful only after its earlier prefix. Preserve that constraint even when sorting candidates globally.

Admission can break exactness

The decision to admit token k must not retrospectively depend on its sampled value through a later continuation. Otherwise high-confidence continuations can select which earlier tokens enter the verifier, biasing the output before the usual acceptance rule runs.

The DSpark appendix gives a counterexample to unrestricted retrospective length search. A residual verifier does not repair this upstream selection bias. In production, the paper uses historical confidence to choose a capacity limit, while current confidence ranks candidate prefixes. This separates capacity selection from the current token realization.

Keep a proof or test for the actual scheduler, not only the token-level verifier. Hardware cliffs, dynamic shapes and delayed predictions can invalidate a naive translation of the mathematical algorithm.

Model-specific recipes

GLM’s KVShare avoids mixing target-produced and draft-produced cache entries across later MTP steps. Its ablation improves accepted length by about 20% in total. It does not isolate IndexShare alone.

The DSpark source reports improved accepted length on Qwen3 4B/8B/14B. Its live V4-Flash comparison reports 60–85% faster per-user generation, and V4-Pro 57–78%, at matched throughput against MTP-1. These are serving-stack observations, not portable multipliers.

Keep the complete report context: deepseek v4 1 flash → · gemma 4 technical report → · glm 5 agentic engineering → · mimo v2 6 scaling rl towards self improvement →.

QSA in MTP: reuse selection, not an unverified answer

A native multi-token prediction (MTP) module also pays for attention selection. Qwen replaces MTP full-attention layers with QSA and reuses top-k indices across prediction steps, following prior GLM work. Repeated selection can be redundant when draft prefixes differ by only a few tokens.

Reusing a mask does not reuse the attention outputs: new queries still compute over selected token representations. It also does not remove target verification, causal masking, acceptance rules or cache rollback. Do not confuse this optimization with cross-layer index sharing, which has different failure modes in a recurrent/global hybrid.

The accepted-length statistic is effectively unchanged in this evaluation. Values above four indicate that this is not simply “number of accepted drafts out of four”; the paper does not fully define its counting convention. Preserve its label rather than converting 4.07 into an acceptance percentage.

Table 4 says four-step speculative decoding. Figure 6's kernel timing instead uses next_n=4, described as three additional MTP steps. Do not silently equate these two conventions. Equal mean accepted length also does not imply equal latency: selection, drafting, verification and memory movement need separate timing.

For an implementation, compare fresh versus reused indices under identical sampling and cache conditions. Track selection cost, accepted length distribution and end-to-end tokens per second. Keep tail-token visibility correct as the causal prefix grows. This is a proposed validation plan, not a released reproduction of their MTP runtime. Selector training and masking → · Measured kernel conditions →.

Qwen3.8-Next architecture report, §2.1.2, Table 4 and Figure 6, pp. 7–9 (26 August 2026) ↗.

Committed tokens versus cached states

A sequence’s logical token list and its processed KV length are different quantities. Sampling a correction or bonus commits a token, but its target KV may not exist until the next forward pass.

Verification can temporarily create states for the full proposed block. Keep states for accepted proposals and remove the rejected suffix. The correction must be processed before later target logits can condition on it. A cache API may handle the pending token on the next call or immediately; document the convention.

The drafter has its own cache and may lag one sampled token behind. Do not assume both caches advance exactly γ entries or already include the correction. Rebuild or advance each according to its actual forward inputs.

Tree KV needs path compaction, not merely a suffix length change. Test cache offsets, position IDs, EOS, sliding-window boundaries and packed requests. Stale states often produce plausible text, which makes them difficult to notice.

Implementation roadmap and tests

Build an exact chain before a learned DSpark system. Tiny compatible causal models make probability and cache errors easier to isolate.

# One position, after obtaining actual sampler distributions p and q.
token = sample(q)
if uniform() < min(1.0, p[token] / q[token]):
    commit(token)
else:
    residual = (p - q).clamp_min(0)
    commit(sample(residual / residual.sum()))
# At rejection, stop the block and invalidate its suffix.

Test p(x)=0, q(x)=0, ties, top-k boundaries, normalization, EOS inside a block and finite precision. Compare greedy tokens exactly and stochastic sequence probabilities statistically or analytically. Matching a few outputs is not enough.

Capture target features from selected layers, tokenizer/chat-template metadata and deployment-domain rollouts. Draft quality changes under temperature, domain and context-length shift. Packed anchors must have isolated attention and no clean-future features.

Start with full-precision probabilities and an ordinary sampler. Add reduced precision only after distributional checks. For the serving scheduler, integrate continuous batching, GPU-resident metadata, stable graph shapes and asynchronous overlap. Profile real engine shapes rather than assuming a smooth capacity curve.

Start with a Markov head before an RNN. Compute unary logits once, sample the first token, look up its predecessor embedding, bias the next slot and repeat. Evaluate calibration separately from discrimination.

Read the benchmark contract

Report accepted-prefix histograms and conditional acceptance by position. Also report verification tokens per committed token, drafter weights/KV memory, cache work and latency by concurrency.

Choose a Pareto frontier over per-user speed, aggregate throughput, tail inter-token latency, memory and operational complexity. The longest accepted block is not necessarily the best system.

Sources and connected notes

Blockwise decoding ↗ · Exact speculative decoding ↗ · Speculative sampling ↗ · SpecInfer ↗ · Medusa ↗ · EAGLE ↗ · EAGLE-2 ↗ · DeepSeek-V3 ↗ · EAGLE-3 ↗ · DFlash ↗ · DSpark ↗ · DFlash implementation ↗