[AutoDeploy][spec-dec] resize_kv_cache sample batch classified as decode-only when max_num_tokens // max_batch_size == 1
@govind-ramnarayan is already working on this.
Since Apr 22, 2026.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Summary
When running Eagle3 one-model (or any other AD spec-dec path that goes through EagleWrapper._forward_with_kv_cache) with a config where max_num_tokens // max_batch_size == 1 (e.g. max_num_tokens=512 and max_batch_size defaulted to the same value), the ResizeKVCache transform builds a sample batch whose heuristic classification is decode-only, which the Eagle wrapper rejects with:
AssertionError: decode without drafting is not supported inside the eagle wrapper
(tensorrt_llm/_torch/auto_deploy/models/custom/modeling_eagle.py:949)
The failure only surfaces for configs that hit the seq_len == 1 edge of set_max_num_tokens_sample — anything with max_num_tokens // max_batch_size > 1 sidesteps it.
Reproduction
tests/integration/defs/examples/test_ad_speculative_decoding.py::test_autodeploy_eagle3_one_model_acceptance_rate[trtllm-torch-cudagraph] and [flashinfer-torch-simple] hit this on 1×H100 when `max_batch_size` is not set (so it defaults to `max_num_tokens=512`). The failure is backend-independent.
Root cause
`ResizeKVCache._apply_to_full_model` (`tensorrt_llm/_torch/auto_deploy/transform/library/kvcache.py:375`):
```python
cm.info.set_max_num_tokens_sample()
if cm._spec_config is not None:
mod(**cm.named_args, cache_seq_interface=cm)
```
`set_max_num_tokens_sample` (`attention_interface.py:951`) computes `seq_len = max_num_tokens // max_batch_size`. When that equals 1, the batch is `[bs, 1]`; the `nest_sequences` heuristic at line 1145–1151 (`(sl_host.flip(0) == 1).cumprod(0).sum()`) then classifies every sequence as decode ⇒ `num_decode=bs, num_extend=0` ⇒ Eagle wrapper's `assert num_decode == 0` fires during resize.
Expected
When `cm._spec_config is not None`, the sample batch used for resize should be an extend-only batch with `seq_len = 1 + max_draft_len` so the forward exercises the same shape the runtime will see.
Proposed fix
In `ResizeKVCache._apply_to_full_model`, replace
```python
cm.info.set_max_num_tokens_sample()
```
with a spec-aware sample, e.g.:
```python
if cm._spec_config is not None:
cm.info.set_capture_batch(max_draft_len=cm._spec_config.max_draft_len)
else:
cm.info.set_max_num_tokens_sample()
```
(or move that branching inside `set_max_num_tokens_sample` itself).
Notes
- Latent on upstream/main — this parametrization of the test was only added to `l0_h100.yml` on branch `gramnarayan/mtp-enable-cudagraph`, so CI never collected it before.
- Test-side workaround (applied on the same branch): set `max_batch_size=128` in the failing test so `seq_len = 512 // 128 = 4 ≥ 1 + max_draft_len(3)`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.