vllm-project / vllm-project/vllm

[Usage]: Does num_speculative_tokens>1 reuse only layer0 for multi-layer MTP (qwen3_5_mtp)?

Open
#52,688 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
92k
Forks
22.3k
PR merge metrics
PR metrics pending

Description

# [Usage/Bug]: Does `num_speculative_tokens > 1` reuse only the first MTP layer for multi-layer MTP models (Qwen3.5-MoE / qwen3_5_mtp)?

### Your current environment

- vLLM 0.18 and 0.22.1 (both reproduce)
- Model: Qwen3.5-MoE family with a native MTP head, `speculative_config={"method":"mtp","num_speculative_tokens":K}`
- TP=2, single node

### The question / suspected issue

When serving a model whose config declares **multiple** MTP layers (`mtp_num_hidden_layers > 1`, i.e. several distinct `mtp.layers.{i}`), and running speculative decoding with `num_speculative_tokens=K>1`, it looks like **all K draft steps reuse `layers[0]` only**, and the other physical MTP layers are loaded but never executed.

Reading the code:

**`vllm/model_executor/models/qwen3_5_mtp.py`** — `Qwen3_5MultiTokenPredictor.forward`:
```python
current_step_idx = spec_step_idx % self.num_mtp_layers
hidden_states, residual = self.layers[current_step_idx](...)
```
So the selected physical layer depends on `spec_step_idx`.

**`vllm/v1/spec_decode/eagle.py`** — the multi-step propose loop (`for token_index in range(self.num_speculative_tokens - 1): ...`) calls `self.model(...)` each iteration but, as far as I can tell, does **not** pass/increment `spec_step_idx`, so `forward` always sees the default `spec_step_idx=0` → `current_step_idx = 0` → `layers[0]` on every draft step.

### Why it matters

For a model trained with N *distinct* MTP layers (each layer trained to predict the (t+i)-th token from the previous layer's hidden state, à la DeepSeek-V3 MTP), only `layers[0]` would actually be used at inference. `layers[1..N-1]` would be dead weight, and per-position acceptance would follow "layer0 applied autoregressively K times" rather than "layer i predicts draft position i". This can meaningfully lower acceptance for multi-layer-trained drafters.

### What I'd like to confirm

1. Is this the intended design (single-layer autoregressive multi-step, ignoring extra MTP layers)?
2. If so, is there a supported path to use *distinct* trained MTP layers per draft position for `qwen3_5_mtp` / `deepseek_mtp`?
3. If it's a bug, `spec_step_idx` presumably needs to be threaded through the `eagle.py` draft loop and incremented per step.

Happy to provide a minimal repro on a public Qwen3.5-MoE checkpoint if useful.

### Before submitting a new issue...

- [x] Searched existing issues; the closest (#52480 drafter shape mismatch, #52469 streamed corruption) describe different failure modes.

Contributor guide

Open the contributing guide

Research direction

Trace Qwen3_5MultiTokenPredictor.forward in vllm/model_executor/models/qwen3_5_mtp.py and the multi-step propose loop in vllm/v1/spec_decode/eagle.py. Check how spec_step_idx is passed across draft iterations, then use the mentioned public Qwen3.5-MoE checkpoint or a minimal repro to verify which MTP layers execute for K>1. Done means confirming the intended behavior or documenting the supported path and evidence for a fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.