[Bug]: MTP accepts no draft tokens with use_kv_cache_manager_v2=false on DSA models (acceptance length 1.0)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
- GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
- Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
- Container:
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25(CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4) - TensorRT-LLM: 1.3.0rc25 (tag
v1.3.0rc25, 785c948); the code paths cited below were also checked onmain@ a6616d6f (2026-09-03) - Backend: PyTorch (
trtllm-serve --backend pytorch)
Who can help?
@NVShreyas (GLM-5 / DSA MTP, #11990) @yizhang-nv (DSA cache managers)
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
GLM-5.3 W4AFP8 on the DeepSeek-V3.2 path (DSA), pp_size 2 --tp_size 4 --ep_size 4, speculative_config: {decoding_type: MTP, max_draft_len: 1}, kv_cache_config: {use_kv_cache_manager_v2: false, enable_block_reuse: true} (V1 KVCacheManager + DSACacheManager; startup logs [DSACacheManager] Indexer k-cache: 12 of 42 local layers own an indexer k-cache, Using Sampler: SpecSampler, CUDA graphs captured with draft_len=1).
Serve real multi-turn agentic traffic (or any long prompts) and read /metrics (JSON iteration stats, enable_iter_perf_stats: true):
iters=1000 drafts=3500 accepted=12 gen_req_iters=3500 ctx_req_iters=0 iter_lat_ms p50=42.5 max=81.8
i.e. every generating request gets one draft token per iteration and essentially none is ever accepted (specDecodingStats.acceptanceLength = 1.0 on every iteration; trtllm_spec_decode_acceptance_length = 1.0 in Prometheus). The same checkpoint, same parallelism and same MTP config with the V2 manager (default use_kv_cache_manager_v2: auto) reports avg_decoded_tokens_per_iter ~1.8 in chat responses, so the draft layer itself is fine.
Outputs remain correct with V1 (200/200 replayed requests clean), so the drafts are simply always rejected: MTP costs a draft forward per step and gains nothing.
Expected behavior
Comparable acceptance with V1 and V2 (or a startup warning/error that MTP is unsupported with the V1 DSA cache manager).
actual behavior
Silent zero acceptance with V1; no warning.
additional notes
- V1 was the only manager that survived our long-context replay long enough to measure (V2 failed three ways: #18660, #18661), so users falling back to V1 lose speculative decoding on DSA models without noticing.
- Guess at the mechanism: the MTP layer's DSA indexer state (
index_share_for_mtp_iteration) or its indexer K-cache is not mirrored/updated for the draft step under the V1DSACacheManager, so the draft attends over stale keys. Not verified.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the PyTorch MTP path and the DSACacheManager/KVCacheManager interaction, focusing on index_share_for_mtp_iteration and the DSA indexer K-cache during the draft step. Reproduce the V1 configuration with enable_iter_perf_stats and compare /metrics acceptanceLength against V2. Done means V1 achieves comparable acceptance or emits a clear unsupported-configuration warning/error, with regression coverage added where the relevant tests are located.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai-infra-agents, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100