vLLM oracle: GPU page fault in ROCm Triton paged-attention fallback (Qwen3.5 GDN, IQ3_XXS, gfx1200)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
Found running ORACLE-VLLM-ROCM-GFX1200-DOCKER (#2961), against the pinned
vLLM oracle (e126687a9a8, ROCm/gfx1200, RX 9060 XT), loading
unsloth/Qwen3.8-27B-GGUF:UD-IQ3_XXS (a Qwen3.5-family Gated-Delta-Net
architecture) plus its mmproj-F16.gguf.
With language_model_only=True (skips multimodal dummy-data profiling) and
enforce_eager=True (skips CUDA-graph capture, whose own "minimal"
profiling KV buffer OOMs independently -- see the row's spec, ## Risks,
for that separate finding), model loading and KV-cache sizing both complete
successfully:
Available KV cache memory: 0.84 GiB
GPU KV cache size: 5,802 tokens, Maximum concurrency for 4,096 tokens per request: 1.42x
Free memory on device (15.82/15.92 GiB) on startup. ... Actual usage is 12.11 GiB
for consumed memory (weights + non-torch), 1.38 GiB for peak activation, and 0.0 GiB
for CUDAGraph memory.
Then, on the very next step:
WARNING [chunked_prefill_paged_decode.py:433] Cannot use ROCm custom paged attention
kernel, falling back to Triton implementation.
Memory access fault by GPU node-1 (Agent handle: 0x247ec020) on address 0x7f229fb44000.
Reason: Page not present or supervisor privilege.
This is a raw HIP-level trap -- no Python traceback names a line, and no
token is ever produced. This is vLLM's/Triton's own ROCm kernel dispatch,
not vllm.cpp code, so it is not fixed in the same flow here; root-causing
a page fault with no Python-level stack needs its own investigation
(AMD_LOG_LEVEL/ROCM_DEBUG, or reducing to a smaller Triton-level repro
off the full 27B model) rather than a guess landed in a hurry. Filed so
.agents/oracles/vllm-gguf-plugin.md has something concrete to point at
when it records this device: this combination has never emitted a token on
gfx1200, which is exactly the fact that keeps that file's gateable = no
correct rather than stale.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read .agents/oracles/vllm-gguf-plugin.md and reproduce the pinned ROCm/gfx1200 setup described here. Use AMD_LOG_LEVEL/ROCM_DEBUG, then reduce the failure to a smaller Triton-level repro if possible. Done means identifying a reproducible kernel-level cause or minimized repro; keep gateable = no until the combination emits a token.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, docker
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100