NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: MTP accepts no draft tokens with use_kv_cache_manager_v2=false on DSA models (acceptance length 1.0)

Open
#18,662 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Speculative Decoding
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • GPU: 8x NVIDIA H200 NVL (SM90; two 4-GPU NVLink islands, cross-island over PCIe/UPI), driver 595.71.05
  • Host: Ubuntu 24.04.4, 2 TiB RAM, Xeon 6747P
  • Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc25 (CUDA 13.2, Python 3.12.3, torch 2.12.0a0+nv26.5, transformers 5.5.4)
  • TensorRT-LLM: 1.3.0rc25 (tag v1.3.0rc25, 785c948); the code paths cited below were also checked on main @ a6616d6f (2026-09-03)
  • Backend: PyTorch (trtllm-serve --backend pytorch)
Who can help?

@NVShreyas (GLM-5 / DSA MTP, #11990) @yizhang-nv (DSA cache managers)

Information
  • The official example scripts
  • My own modified scripts
Tasks
  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)
Reproduction

GLM-5.3 W4AFP8 on the DeepSeek-V3.2 path (DSA), pp_size 2 --tp_size 4 --ep_size 4, speculative_config: {decoding_type: MTP, max_draft_len: 1}, kv_cache_config: {use_kv_cache_manager_v2: false, enable_block_reuse: true} (V1 KVCacheManager + DSACacheManager; startup logs [DSACacheManager] Indexer k-cache: 12 of 42 local layers own an indexer k-cache, Using Sampler: SpecSampler, CUDA graphs captured with draft_len=1).

Serve real multi-turn agentic traffic (or any long prompts) and read /metrics (JSON iteration stats, enable_iter_perf_stats: true):

iters=1000 drafts=3500 accepted=12 gen_req_iters=3500 ctx_req_iters=0 iter_lat_ms p50=42.5 max=81.8

i.e. every generating request gets one draft token per iteration and essentially none is ever accepted (specDecodingStats.acceptanceLength = 1.0 on every iteration; trtllm_spec_decode_acceptance_length = 1.0 in Prometheus). The same checkpoint, same parallelism and same MTP config with the V2 manager (default use_kv_cache_manager_v2: auto) reports avg_decoded_tokens_per_iter ~1.8 in chat responses, so the draft layer itself is fine.

Outputs remain correct with V1 (200/200 replayed requests clean), so the drafts are simply always rejected: MTP costs a draft forward per step and gains nothing.

Expected behavior

Comparable acceptance with V1 and V2 (or a startup warning/error that MTP is unsupported with the V1 DSA cache manager).

actual behavior

Silent zero acceptance with V1; no warning.

additional notes
  • V1 was the only manager that survived our long-context replay long enough to measure (V2 failed three ways: #18660, #18661), so users falling back to V1 lose speculative decoding on DSA models without noticing.
  • Guess at the mechanism: the MTP layer's DSA indexer state (index_share_for_mtp_iteration) or its indexer K-cache is not mirrored/updated for the draft step under the V1 DSACacheManager, so the draft attends over stale keys. Not verified.
Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the PyTorch MTP path and the DSACacheManager/KVCacheManager interaction, focusing on index_share_for_mtp_iteration and the DSA indexer K-cache during the draft step. Reproduce the V1 configuration with enable_iter_perf_stats and compare /metrics acceptanceLength against V2. Done means V1 achieves comparable acceptance or emits a clear unsupported-configuration warning/error, with regression coverage added where the relevant tests are located.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai-infra-agents, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.