Bringing DSpark to vLLM Metal

Steven Wang (@StevenWang-CY) led this work and authored the core DSpark series. I was co-author on those patches and authored the follow-ups on prefix caching, configuration validation, and candidate-limit handling. This is a joint retrospective.

DSpark is a block speculative-decoding method: a lightweight draft model proposes a block of K tokens, and the target model verifies the whole block in one parallel pass. On a discrete GPU the appeal is obvious — trade a little extra compute for fewer memory-bound decode steps. On Apple Silicon the trade is subtler. Unified memory removes the host-to-device copies that dominate discrete setups, but it also makes every extra forward pass compete for the same bandwidth and the same scheduler budget. A draft path that is merely correct can easily be net-negative.

That was the starting point for DSpark in vLLM Metal. The feature landed as an experimental, greedy-only serving path for matched Qwen3 target/draft pairs. The first matched benchmark was negative: under the same synchronous scheduler, DSpark was slower than target-only. The later Speculators benchmark, against the asynchronous serving baseline, showed a 2.29× median throughput gain in BF16 short C1 and 1.16× in Q4 — with mixed long-context behavior and correctness still open. This is how it was built, and why both results are true.

The part that is easy to underestimate: draft KV ownership

The earlier draft-model investigation had already measured the shape of the problem (#482): at 8K context, the first propose of a new request spent about 4.6 seconds re-prefilling the prompt into the draft KV, and warm proposing still cost roughly 1.7× a complete target-only decode step. DSpark had the same structural issue — it recomputed the full prefix for every draft block. The cause was not a slow kernel. The draft model owned a private KV allocator with no content-based reuse, so every new request re-prefilled its entire prompt, and every propose round recomputed KV that the previous round had already written.

The redesign moved draft KV under vLLM's scheduler ownership. That sounds like bookkeeping. It is the feature.

In the merged design (#977), the drafter's committed KV is registered as a scheduler cache group, sharing the paged backbone DFlash already uses. The layout has to match DSpark's block semantics exactly: K query slots including the slot-zero anchor prediction, sequential Markov corrections, and an optional confidence head. DFlash uses K+1 slots; DSpark uses exactly K. At K=1 the path collapses to ordinary paged decode; larger blocks keep explicit bidirectional attention over the draft window.

Two invariants matter more than the data structures:

1. Only verified target features become committed context. Temporary draft KV is overwritten after verification. A rejected draft must not leave reusable prefix state behind.
2. A target cache hit is not automatically a draft cache hit. Prefix reuse requires valid committed data in every required cache group. If the target can skip a forward, the missing draft features still have to be produced — a coordinated recomputation problem, not a cache lookup.

Bindings also require distinct layers from one scheduler group with matching KV block size and precision. Separate cache groups can overlay physical memory, but they cannot share this single-table forward. Getting those constraints into validation — rather than into a runtime assertion deep inside a Metal kernel — was a recurring theme.

From "it runs" to "we can measure it"

The next milestone (#986) wired the paged adapter into Metal's synchronous runner for plain greedy requests. Prefix caching was deliberately disabled at this stage; the point was to prove the lifecycle, not to sell a speedup.

We tested with a real pair: mlx-community/Qwen3-4B-4bit as the target and deepseek-ai/dspark_qwen3_4b_block7 as the draft. The first end-to-end result was encouraging in the way that matters most for correctness: 32/32 output IDs matched target-only serving, with 28 draft tokens verified and accepted. The suite grew to 3,963 passing tests, covering both verification layouts, cache pressure, page reclamation, request reuse, width transitions, fallback requests, and stop/EOS handling.

Then came the first matched HTTP benchmark (#999), and the honest answer: under the same synchronous scheduler, DSpark was slower than target-only serving on the tested workload.

The first matched benchmark as a table, including the ordinary draft path:

Apple M5 Max, Qwen3-4B-4bit target, synchronous serving, two reversed-order repeats, 40 prompts, 64 output tokens. Each cell retains both repeats; the desktop had paging activity, so this is not an isolated hardware result.

The memory story was more interesting. DSpark's scheduler KV backing was 5.97 GiB versus 8.63 GiB for target-only at the same memory fraction — it gives up cache capacity to the draft path but keeps more of it than an ordinary draft model. Worker RSS peak was higher (5.52 GiB vs 2.96 GiB), and those accounting domains should not be added together. At that point the headline was not "DSpark is faster." It was "we now have the harness to find out where the time goes."

Apple M5 Max, same memory fraction. MLX allocation, worker RSS, and reserved KV are different accounting domains; do not add them together. Source: #999.

That result was not the end of the story. The matched comparison isolates the algorithm; it does not answer what a user gets when they turn DSpark on in place of the async serving path. The Speculators benchmark below asks that second question.

Making the draft cheaper

Two optimizations moved the needle without changing the correctness contract.

Top-K Markov drafting (#1003). DSpark's Markov correction was projected across the full vocabulary at every draft position. The new path selects candidates from the base logits first, then applies the sequential correction only to those projection rows. The target still verifies against its full vocabulary. On the 8B pair, head-only time at batch one fell from 3.59–3.66 ms to 2.93–2.97 ms with dspark_draft_topk: 64; matched HTTP runs at concurrency one improved 6–12%. At batch four there was no consistent gain, and both configurations remained slower than target-only. Candidate limiting can change proposals and acceptance, so it is a tuning knob, not a free win.

Q4 draft weights (#1041). The floating-point draft weights for the Qwen3-8B pair consumed 4.42 GiB. An opt-in runtime Q4 conversion cut the tensor payload to 2.29 GiB — 48% less — leaving more of the device budget for scheduler-owned KV. It requires a floating-point checkpoint; pre-quantized drafts are not supported. Again: memory is not speed. The benchmark is the arbiter.

The follow-ups I owned were less glamorous but necessary for a feature people can actually run. Prefix caching (#1044) had been rejected at startup for both DSpark and DFlash; new and resumed requests can now reuse the scheduler's common target/draft prefix and compute only the suffix. Configuration validation (#1048) rejects KV offloading with block drafting before weights are loaded — block drafting needs both target and draft cache groups, while the Metal offloader supports a single full-attention group. And #1049 accepts integral candidate limits from Python callers, including NumPy integers, instead of raising ValueError on a perfectly valid np.int64(8).

Those changes are the bridge to the Speculators result: Q4 shrinks the draft footprint, top-K cuts the head cost, and prefix caching makes repeated requests cheaper. The matched benchmark we had was still negative. What changed the serving picture was the Red Hat Qwen3-4B Speculators checkpoint and a comparison against the async baseline rather than against same-scheduler target-only.

The Speculators benchmark: 2.29× on short prompts

The first matched benchmark asked whether DSpark beats target-only under the same synchronous scheduler. The serving question is different. The baseline most users run is asynchronous target-only serving, while DSpark currently requires synchronous scheduling. The Speculators benchmark measures that whole configuration: DSpark on, synchronous, against DSpark off, asynchronous.

On the Red Hat AI Qwen3-4B Speculators checkpoint, with short prompts and a single request, DSpark's median output throughput was 2.29× the async baseline in BF16, and 1.16× in Q4. The gain is large in that setting but narrow in scope. BF16 long-context C4 was 1.06×; Q4 long-context C4 was 0.79×. BF16 short C1 also paid 1.45× p95 first-token latency — worse — even as mean token time fell to 0.42×. EAL, the expected acceptance length, was 4.00 on BF16 short C1 and 3.82 on Q4 short C1, falling to 3.43 and 3.31 on long C4.

Speculators benchmark · Apple M5 Max · median ratios

DSpark vs async serving

Output throughput · change vs DSpark-off async baseline · positive is better

Throughput p95 first token Mean token time p95 completion

DSpark runs with synchronous scheduling; the baseline is asynchronous serving without DSpark. This is a system-level comparison, not a matched-scheduler A/B. Long-context results are mixed.

The same benchmark as a table:

Red Hat AI Qwen3-4B Speculators, Apple M5 Max. Median ratios of DSpark-on (synchronous) to DSpark-off (asynchronous) serving; for latency, below 1.00× is better. EAL is expected acceptance length. The source figure reports medians and variability; long-context behavior is mixed and output-consistency failures remain. Source: the release benchmark figure; see #1092 and #1136.

Read the 2.29× as a system-level, short-prompt, single-request result, not a universal speedup. DSpark-on-sync is being compared with DSpark-off-async. The long-context rows are the counterexamples; the p95 first-token number is the latency counterexample even within the short C1 setting. And a throughput ratio is not a correctness result — the consistency failures below still have to clear.

Correctness is the feature

DSpark is experimental for a reason. The parity tool compares fresh greedy target-only and speculative runs and reports exact matches separately from top-K matches. On the shared 40-prompt, 32-token native mlx-lm comparison, DSpark matched same-batch target-only sequences exactly on 29/40 and 32/40 prompts in the 4B runs; the 8B runs showed similar, workload-dependent numbers. Bitwise serving losslessness is not established.

The Speculators benchmark does not change this gate. Its 2.29× is a throughput result, not an output-equality result; the release announcement still lists output-consistency failures as unresolved.

The reduced-precision gates are explicit and unresolved. In FP16, all 231 proposal IDs match between native paged and dense paths, but six one-token-prefix cases exceed the logit tolerance (atol=0.015, rtol=0.02; max absolute error 0.04004). In BF16, 41/42 cases pass; a near tie at batch 2, context 1, K=7 changes a proposal and later Markov corrections. Those failures remain failures. We did not widen the tolerances to make the dashboard green.

That discipline is the most valuable thing I took from the collaboration. A speculative-decoding feature has three separable claims: it runs, it produces the right tokens, and it is faster. They need separate evidence, and the first two have to hold before the third is worth measuring.

Where it stands

DSpark now supports matched Qwen3 target/draft pairs, both DeepSpec and Qwen3 Speculators checkpoints, greedy serving, prefix caching, top-K Markov candidates, and opt-in Q4 draft weights. The Speculators benchmark is the first positive serving result in this story: 2.29× median throughput in BF16 short C1 against the async baseline, 1.16× in Q4, with mixed long-context behavior. A minimal serving setup looks like this:

vllm serve mlx-community/Qwen3-4B-4bit \
    --max-model-len 2048 \
    --no-async-scheduling \
    --enable-prefix-caching \
    --speculative-config '{
      "method": "dspark",
      "model": "deepseek-ai/dspark_qwen3_4b_block7",
      "num_speculative_tokens": 7
    }'

Sampling, penalties, grammar constraints, and sample logprobs fall back to target-only generation. Asynchronous scheduling, LoRA, TurboQuant, and KV offloading are not supported with DSpark. The work in flight is exactly what you would expect: adaptive planning, sampled verification, asynchronous scheduling, better prefix reuse, cheaper verification attention (#1103), fused draft KV gathering (#1136), skipping drafts after EOS and stop tokens (#1203), and CI that checks acceptance after prefix reuse (#1184).

The first matched benchmark did not hand us a speedup. It handed us the harness that later found one — and showed us its edges. That is the state DSpark on Metal needed to reach.

Thanks to Steven for leading the series, and to the vLLM Metal maintainers for the review pressure — especially the insistence on an explicit block-mask interface, a prefix-cache-disabled experimental stage, and keeping every qualification failure visible.