NVIDIA / NVIDIA/TensorRT-LLM

[AutoDeploy][spec-dec] resize_kv_cache sample batch classified as decode-only when max_num_tokens // max_batch_size == 1

Open
#13,348 1 comment 0 reactions 1 assignee View on GitHub

@govind-ramnarayan is already working on this.

Since Apr 22, 2026.

Speculative Decoding
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Summary

When running Eagle3 one-model (or any other AD spec-dec path that goes through EagleWrapper._forward_with_kv_cache) with a config where max_num_tokens // max_batch_size == 1 (e.g. max_num_tokens=512 and max_batch_size defaulted to the same value), the ResizeKVCache transform builds a sample batch whose heuristic classification is decode-only, which the Eagle wrapper rejects with:

AssertionError: decode without drafting is not supported inside the eagle wrapper

(tensorrt_llm/_torch/auto_deploy/models/custom/modeling_eagle.py:949)

The failure only surfaces for configs that hit the seq_len == 1 edge of set_max_num_tokens_sample — anything with max_num_tokens // max_batch_size > 1 sidesteps it.

Reproduction

tests/integration/defs/examples/test_ad_speculative_decoding.py::test_autodeploy_eagle3_one_model_acceptance_rate[trtllm-torch-cudagraph] and [flashinfer-torch-simple] hit this on 1×H100 when `max_batch_size` is not set (so it defaults to `max_num_tokens=512`). The failure is backend-independent.

Root cause

`ResizeKVCache._apply_to_full_model` (`tensorrt_llm/_torch/auto_deploy/transform/library/kvcache.py:375`):

```python
cm.info.set_max_num_tokens_sample()
if cm._spec_config is not None:
mod(**cm.named_args, cache_seq_interface=cm)
```

`set_max_num_tokens_sample` (`attention_interface.py:951`) computes `seq_len = max_num_tokens // max_batch_size`. When that equals 1, the batch is `[bs, 1]`; the `nest_sequences` heuristic at line 1145–1151 (`(sl_host.flip(0) == 1).cumprod(0).sum()`) then classifies every sequence as decode ⇒ `num_decode=bs, num_extend=0` ⇒ Eagle wrapper's `assert num_decode == 0` fires during resize.

Expected

When `cm._spec_config is not None`, the sample batch used for resize should be an extend-only batch with `seq_len = 1 + max_draft_len` so the forward exercises the same shape the runtime will see.

Proposed fix

In `ResizeKVCache._apply_to_full_model`, replace
```python
cm.info.set_max_num_tokens_sample()
```
with a spec-aware sample, e.g.:
```python
if cm._spec_config is not None:
cm.info.set_capture_batch(max_draft_len=cm._spec_config.max_draft_len)
else:
cm.info.set_max_num_tokens_sample()
```
(or move that branching inside `set_max_num_tokens_sample` itself).

Notes
  • Latent on upstream/main — this parametrization of the test was only added to `l0_h100.yml` on branch `gramnarayan/mtp-enable-cudagraph`, so CI never collected it before.
  • Test-side workaround (applied on the same branch): set `max_batch_size=128` in the failing test so `seq_len = 512 // 128 = 4 ≥ 1 + max_draft_len(3)`.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.