intel / intel/llm-scaler

[vllm][compute-runtime] Host RAM ballooning with tp>1, bug in latest compute-runtime versions

Open
#618 1 comment 0 reactions 1 assignee Claimed by @gc-fu View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

### Summary

`intel/llm-scaler-vllm:0.21.0-b3.1` bundles compute-runtime `26.14.37833.4`. That version is past the regression window described in [intel/compute-runtime#953](https://github.com/intel/compute-runtime/issues/953) ("26.09+"), where every large device allocation on one GPU gets an eagerly-pinned host-RAM mirror registered on the peer GPU's context for P2P bookkeeping. In that issue, maintainer **jwarchul**:

- 2026-07-16: suggested the debug-key workaround `NEOReadDebugKeys=1 EnableDeferBacking=0`
- 2026-08-03: reversed that guidance — debug keys are "intended strictly for debugging purposes and should not be used in production environments" — and recommended **downgrading to compute-runtime 26.05** as "the safest option for production" until a proper fix ships.

I reproduced the ballooning on this image with a plain TP=2 vLLM serve of `Qwen/Qwen3.8-27B-FP8` across 2x Arc Pro B70 (no speculative decoding, no MTP — the simplest possible config), then re-ran with the debug-key flag as an A/B test.

### Platform
- Ryzen 9900X, ASUS ProArt Creator X870 WiFi, 128GB DDR5-5400 (4x32GB dual-rank UDIMM)
- 2x Intel Arc Pro B70, PCIe Gen5 x8/x8, P2P DMA independently verified via `ze_peer` (28.5 GB/s per direction, 56.4 GB/s bidirectional — full Gen5 x8/x8 ceiling, so this isn't a P2P-capability problem, it's the bookkeeping-pinning behavior itself)
- Host compute-runtime: `26.27.39122.12`; image-bundled compute-runtime: `26.14.37833.4`

### Repro command
```
vllm serve Qwen/Qwen3.8-27B-FP8 \
--tensor-parallel-size 2 --dtype float16 --trust-remote-code --language-model-only \
--max-model-len 8192 --max-num-seqs 128 --gpu-memory-utilization 0.90 \
--tool-call-parser qwen3_xml --enable-auto-tool-choice --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking": false}'
```

### Results — before vs. after, idle server, zero requests served except one functional check

| | Baseline (no flag) | With `NEOReadDebugKeys=1 EnableDeferBacking=0` |
|---|---|---|
| Host RAM `used`, before container start | 9.2 GiB | 9.7 GiB |
| Host RAM `used`, idle+healthy | **76 GiB** | **19 GiB** |
| Delta | **+67 GiB** | **+9 GiB** |
| GPU0 / GPU1 VRAM | 30.7 / 30.3 GiB | 31.4 / 31.0 GiB |

VRAM usage is essentially identical between the two runs (model correctly loaded and sharded either way), and the flagged run serves a correct completion — so this isn't a correctness or capability tradeoff, it's purely the host-RAM bookkeeping bug getting turned off. The ~67 GiB pinned in the baseline run is also, separately, invisible to `docker stats` (9.1 GiB in-cgroup) and to summed process RSS across the whole host (22.7 GB) — matches this issue's and intel/compute-runtime#968's description of the memory not being attributable via normal Linux accounting.

### Ask

Given compute-runtime's own maintainer says the debug-key flag isn't production-safe and recommends pinning to 26.05 instead, could the image pin/downgrade its bundled compute-runtime to a pre-regression build (26.05, per the maintainer's recommendation, or whatever version llm-scaler validates as unaffected) rather than shipping 26.14 unpinned? Multi-GPU TP inference is the primary use case for this image, and this bug currently makes any TP≥2 config with meaningfully-sized models eat 50-70+ GiB of host RAM for no visible reason, which is a hard blocker on memory-constrained hosts and very difficult for users to self-diagnose (as intel/compute-runtime#968 and my own dead-end debugging before finding #953 both show).

Happy to test a repin against 26.05 (or any candidate build) on this hardware if useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.