vllm-project / vllm-project/vllm
[Usage]: Does num_speculative_tokens>1 reuse only layer0 for multi-layer MTP (qwen3_5_mtp)?
- Dominant language
- Python
- Stars
- 92k
- Forks
- 22.3k
- PR merge metrics
- PR metrics pending
Description
# [Usage/Bug]: Does `num_speculative_tokens > 1` reuse only the first MTP layer for multi-layer MTP models (Qwen3.5-MoE / qwen3_5_mtp)?
### Your current environment
- vLLM 0.18 and 0.22.1 (both reproduce)
- Model: Qwen3.5-MoE family with a native MTP head, `speculative_config={"method":"mtp","num_speculative_tokens":K}`
- TP=2, single node
### The question / suspected issue
When serving a model whose config declares **multiple** MTP layers (`mtp_num_hidden_layers > 1`, i.e. several distinct `mtp.layers.{i}`), and running speculative decoding with `num_speculative_tokens=K>1`, it looks like **all K draft steps reuse `layers[0]` only**, and the other physical MTP layers are loaded but never executed.
Reading the code:
**`vllm/model_executor/models/qwen3_5_mtp.py`** — `Qwen3_5MultiTokenPredictor.forward`:
```python
current_step_idx = spec_step_idx % self.num_mtp_layers
hidden_states, residual = self.layers[current_step_idx](...)
```
So the selected physical layer depends on `spec_step_idx`.
**`vllm/v1/spec_decode/eagle.py`** — the multi-step propose loop (`for token_index in range(self.num_speculative_tokens - 1): ...`) calls `self.model(...)` each iteration but, as far as I can tell, does **not** pass/increment `spec_step_idx`, so `forward` always sees the default `spec_step_idx=0` → `current_step_idx = 0` → `layers[0]` on every draft step.
### Why it matters
For a model trained with N *distinct* MTP layers (each layer trained to predict the (t+i)-th token from the previous layer's hidden state, à la DeepSeek-V3 MTP), only `layers[0]` would actually be used at inference. `layers[1..N-1]` would be dead weight, and per-position acceptance would follow "layer0 applied autoregressively K times" rather than "layer i predicts draft position i". This can meaningfully lower acceptance for multi-layer-trained drafters.
### What I'd like to confirm
1. Is this the intended design (single-layer autoregressive multi-step, ignoring extra MTP layers)?
2. If so, is there a supported path to use *distinct* trained MTP layers per draft position for `qwen3_5_mtp` / `deepseek_mtp`?
3. If it's a bug, `spec_step_idx` presumably needs to be threaded through the `eagle.py` draft loop and incremented per step.
Happy to provide a minimal repro on a public Qwen3.5-MoE checkpoint if useful.
### Before submitting a new issue...
- [x] Searched existing issues; the closest (#52480 drafter shape mismatch, #52469 streamed corruption) describe different failure modes.
Contributor guide
Research direction
Trace Qwen3_5MultiTokenPredictor.forward in vllm/model_executor/models/qwen3_5_mtp.py and the multi-step propose loop in vllm/v1/spec_decode/eagle.py. Check how spec_step_idx is passed across draft iterations, then use the mentioned public Qwen3.5-MoE checkpoint or a minimal repro to verify which MTP layers execute for K>1. Done means confirming the intended behavior or documenting the supported path and evidence for a fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100