vllm-project / vllm-project/vllm

[Bug][ROCm] GLM-5.3-Flash indexer block-table mismatch on gfx942

Open
#56,380 1 comment 0 reactions 0 assignees View on GitHub
glm rocm
Dominant language
Python
Stars
91.8k
Forks
22.2k
PR merge metrics
PR metrics pending

Description

## Environment

- 8x MI308X (gfx942), ROCm 7.2.3
- vLLM `0.28.1rc1.dev628+g2a02f6efe.rocm723`, `amd_aiter==0.1.21.post2`, `flydsl==0.3.2`
- GLM-5.3-Flash, TP8, bf16 KV, prefix caching enabled
- `index_kpool=4`, `index_topk=2048`, `max_num_batched_tokens=8192`

## 1. Indexer block-table granularity mismatch

Long-prompt requests completed, but recall near the end degraded. The indexer cache uses 128-token storage pages (32 pooled entries), while the block table uses 640-token manager blocks:

- `allocate_kv_cache` honors `MLAAttentionSpec.storage_block_size`.
- `prepare_kernel_block_sizes` does not, leaving manager block IDs where physical page IDs are needed.

At `max_model_len=262144`, the 410-column table can address only `410 * 32 * 4 = 52480` token positions when interpreted by the pool-page consumer. This is a layout mismatch, not a model context-length limit.

Observed before the fix:

```text
cache_shape=(91780, 32, 132) block_table_shape=(1, 410)
required_pooled_entries=40160
last_query_position=160639 max_selected_token_index=52479
```

Outside the table width, the gather masks the block-ID read and writes a zero scale. Inside the width, manager block IDs can name the wrong physical pages; this is not a fixed token offset.

PR #56381 makes the table use storage-page granularity. At the **same 262144-token setting**, it becomes `(1, 2050)`; manager block `b` expands to pages `5*b + [0, 1, 2, 3, 4]`.

## 2. Separate kpool-tail slot-mapping fault

A 7680-token prefill step also produced GPU memory-access faults. With `HIP_LAUNCH_BLOCKING=1 PYTHONFAULTHANDLER=1`, the stack pointed to `_compute_slot_mappings_kernel`.

`KpoolTailSpec` reserves one circular block per request, but the generic mapper uses `position // 4` against a 32-column row. Its mapping should be left to `KpoolTailMetadataBuilder`. Fault occurrence depends on allocation; 7680 is an observed step size, not a universal threshold.

**PR #56381 addresses only problem 1.**

## Reproduction and validation scope

For problem 1, compare the same configuration before/after the PR with a roughly 160K-token prompt. Log the indexer cache shape, block-table shape/IDs, and the final prefill row's selected indices. Expected table entries are physical page IDs.

The model runs also used local fixes for FlyDSL `shrui(int)`, FP8 pack zero-extension, and problem 2. A stock build may fail on those paths first.

Separate 524288-window runs completed requests up to about 519K prompt tokens. These were diagnostic probes, not accuracy benchmarks: SEAL searched final and reasoning text; ECHO matched a 24-character substring. A self-contained reproducer and automated regression test are still needed.

## Related work

#55219 replaces this cache layout and removes `storage_block_size` and `KpoolTailSpec`. It may supersede the narrow fix in #56381; its description reports no ROCm kernel run.

AI assistance was used for investigation, implementation, and this report.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.