[vllm][compute-runtime] Host RAM ballooning with tp>1, bug in latest compute-runtime versions
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
### Summary
`intel/llm-scaler-vllm:0.21.0-b3.1` bundles compute-runtime `26.14.37833.4`. That version is past the regression window described in [intel/compute-runtime#953](https://github.com/intel/compute-runtime/issues/953) ("26.09+"), where every large device allocation on one GPU gets an eagerly-pinned host-RAM mirror registered on the peer GPU's context for P2P bookkeeping. In that issue, maintainer **jwarchul**:
- 2026-07-16: suggested the debug-key workaround `NEOReadDebugKeys=1 EnableDeferBacking=0`
- 2026-08-03: reversed that guidance — debug keys are "intended strictly for debugging purposes and should not be used in production environments" — and recommended **downgrading to compute-runtime 26.05** as "the safest option for production" until a proper fix ships.
I reproduced the ballooning on this image with a plain TP=2 vLLM serve of `Qwen/Qwen3.8-27B-FP8` across 2x Arc Pro B70 (no speculative decoding, no MTP — the simplest possible config), then re-ran with the debug-key flag as an A/B test.
### Platform
- Ryzen 9900X, ASUS ProArt Creator X870 WiFi, 128GB DDR5-5400 (4x32GB dual-rank UDIMM)
- 2x Intel Arc Pro B70, PCIe Gen5 x8/x8, P2P DMA independently verified via `ze_peer` (28.5 GB/s per direction, 56.4 GB/s bidirectional — full Gen5 x8/x8 ceiling, so this isn't a P2P-capability problem, it's the bookkeeping-pinning behavior itself)
- Host compute-runtime: `26.27.39122.12`; image-bundled compute-runtime: `26.14.37833.4`
### Repro command
```
vllm serve Qwen/Qwen3.8-27B-FP8 \
--tensor-parallel-size 2 --dtype float16 --trust-remote-code --language-model-only \
--max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.90 \
--tool-call-parser qwen3_xml --enable-auto-tool-choice --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": false}'
```
### Results — before vs. after, idle server, zero requests served except one functional check
| | Baseline (no flag) | With `NEOReadDebugKeys=1 EnableDeferBacking=0` |
|---|---|---|
| Host RAM `used`, before container start | 9.2 GiB | 9.7 GiB |
| Host RAM `used`, idle+healthy | **76 GiB** | **19 GiB** |
| Delta | **+67 GiB** | **+9 GiB** |
| GPU0 / GPU1 VRAM | 30.7 / 30.3 GiB | 31.4 / 31.0 GiB |
VRAM usage is essentially identical between the two runs (model correctly loaded and sharded either way), and the flagged run serves a correct completion — so this isn't a correctness or capability tradeoff, it's purely the host-RAM bookkeeping bug getting turned off. The ~67 GiB pinned in the baseline run is also, separately, invisible to `docker stats` (9.1 GiB in-cgroup) and to summed process RSS across the whole host (22.7 GB) — matches this issue's and intel/compute-runtime#968's description of the memory not being attributable via normal Linux accounting.
### Ask
Given compute-runtime's own maintainer says the debug-key flag isn't production-safe and recommends pinning to 26.05 instead, could the image pin/downgrade its bundled compute-runtime to a pre-regression build (26.05, per the maintainer's recommendation, or whatever version llm-scaler validates as unaffected) rather than shipping 26.14 unpinned? Multi-GPU TP inference is the primary use case for this image, and this bug currently makes any TP≥2 config with meaningfully-sized models eat 50-70+ GiB of host RAM for no visible reason, which is a hard blocker on memory-constrained hosts and very difficult for users to self-diagnose (as intel/compute-runtime#968 and my own dead-end debugging before finding #953 both show).
Happy to test a repin against 26.05 (or any candidate build) on this hardware if useful.
Contributor guide
Assessment
This issue has not been assessed yet.