vllm-project / vllm-project/vllm
[Bug][ROCm] GLM-5.3-Flash indexer block-table mismatch on gfx942
- Dominant language
- Python
- Stars
- 91.8k
- Forks
- 22.2k
- PR merge metrics
- PR metrics pending
Description
## Environment
- 8x MI308X (gfx942), ROCm 7.2.3
- vLLM `0.28.1rc1.dev628+g2a02f6efe.rocm723`, `amd_aiter==0.1.21.post2`, `flydsl==0.3.2`
- GLM-5.3-Flash, TP8, bf16 KV, prefix caching enabled
- `index_kpool=4`, `index_topk=2048`, `max_num_batched_tokens=8192`
## 1. Indexer block-table granularity mismatch
Long-prompt requests completed, but recall near the end degraded. The indexer cache uses 128-token storage pages (32 pooled entries), while the block table uses 640-token manager blocks:
- `allocate_kv_cache` honors `MLAAttentionSpec.storage_block_size`.
- `prepare_kernel_block_sizes` does not, leaving manager block IDs where physical page IDs are needed.
At `max_model_len=262144`, the 410-column table can address only `410 * 32 * 4 = 52480` token positions when interpreted by the pool-page consumer. This is a layout mismatch, not a model context-length limit.
Observed before the fix:
```text
cache_shape=(91780, 32, 132) block_table_shape=(1, 410)
required_pooled_entries=40160
last_query_position=160639 max_selected_token_index=52479
```
Outside the table width, the gather masks the block-ID read and writes a zero scale. Inside the width, manager block IDs can name the wrong physical pages; this is not a fixed token offset.
PR #56381 makes the table use storage-page granularity. At the **same 262144-token setting**, it becomes `(1, 2050)`; manager block `b` expands to pages `5*b + [0, 1, 2, 3, 4]`.
## 2. Separate kpool-tail slot-mapping fault
A 7680-token prefill step also produced GPU memory-access faults. With `HIP_LAUNCH_BLOCKING=1 PYTHONFAULTHANDLER=1`, the stack pointed to `_compute_slot_mappings_kernel`.
`KpoolTailSpec` reserves one circular block per request, but the generic mapper uses `position // 4` against a 32-column row. Its mapping should be left to `KpoolTailMetadataBuilder`. Fault occurrence depends on allocation; 7680 is an observed step size, not a universal threshold.
**PR #56381 addresses only problem 1.**
## Reproduction and validation scope
For problem 1, compare the same configuration before/after the PR with a roughly 160K-token prompt. Log the indexer cache shape, block-table shape/IDs, and the final prefill row's selected indices. Expected table entries are physical page IDs.
The model runs also used local fixes for FlyDSL `shrui(int)`, FP8 pack zero-extension, and problem 2. A stock build may fail on those paths first.
Separate 524288-window runs completed requests up to about 519K prompt tokens. These were diagnostic probes, not accuracy benchmarks: SEAL searched final and reasoning text; ECHO matched a 24-character substring. A self-contained reproducer and automated regression test are still needed.
## Related work
#55219 replaces this cache layout and removes `storage_block_size` and `KpoolTailSpec`. It may supersede the narrow fix in #56381; its description reports no ROCm kernel run.
AI assistance was used for investigation, implementation, and this report.
Contributor guide
Assessment
This issue has not been assessed yet.