[vllm] Host RAM leak (~25 MB/request) in pipeline-parallel send path on dual Arc Pro B60 without P2P; tensor-parallel is clean (0.21.0-b1)
- Dominant language
- C++
- Stars
- 529
- Forks
- 80
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 38
Description
## Summary
With `intel/llm-scaler-vllm:0.21.0-b1` serving Qwen3.6-35B-A3B (MoE) with **pipeline-parallel-size 2** on 2× Arc Pro B60 (no common PCIe upstream bridge → no P2P; all inter-GPU traffic is host-staged), the **first PP rank leaks host RAM proportional to processed tokens**: ~23–40 MB per request, never freed, even when the engine is fully idle (0 running/waiting requests, KV cache usage 0%).
An identical test with **tensor-parallel-size 2 shows no leak** — the issue is specific to the pt2pt send/recv path, not the all_reduce path.
## Environment
- 2× Intel Arc Pro B60 24 GB (BMG G21), **no shared PCIe upstream bridge** (kernel log: "cannot be used for peer-to-peer DMA"), Alder Lake-S host, 62 GB RAM
- Ubuntu 24.04, kernel 7.0.0-28 (xe driver), Docker
- Image: `intel/llm-scaler-vllm:0.21.0-b1` (vLLM `v0.21.1.dev0+gad7125a43`)
- Model: `Qwen/Qwen3.6-35B-A3B`, runtime fp8 (`--quantization fp8 --dtype float16`), torch.compile enabled (no `--enforce-eager`), `--max-model-len 8192 --max-num-seqs 32`
- Env required for this topology (cf. #550): `CCL_ZE_IPC_EXCHANGE=pidfd FI_PROVIDER=tcp FI_TCP_IFACE=lo CCL_TOPO_P2P_ACCESS=0 CCL_SEND=direct CCL_RECV=direct VLLM_WORKER_MULTIPROC_METHOD=spawn`
## Reproduction
1. Serve with `--pipeline-parallel-size 2 --tensor-parallel-size 1`.
2. Send 40 chat completions (~230 prompt tokens, 128 output tokens, concurrency 8, non-streaming).
3. Compare per-process host RSS before/after; engine idle at both measurement points.
## Measurements (per-process RSS delta after the same 40 requests)
| process | PP=2 | TP=2 (identical test) |
|---|---|---|
| Worker rank 0 | **+924 MB** | +15 MB |
| Worker rank 1 | +16 MB | +15 MB |
| EngineCore | +0 | +0 |
| APIServer | +1 MB | +1 MB |
| container anon total | 10.77 → 11.63 GiB | 12.59 → **12.59 GiB** |
Additional observations:
- The leak scales with tokens processed, not with time: a production run grew 11 GiB → 24.1 GiB over 350 requests (~1000 tokens in / up to 2500 out each) and then **stayed flat while idle** for 10+ minutes (0 requests, KV 0%). Nothing is ever returned.
- Rank 0 is the **send side** of the large activation transfers (PP stage 0 → stage 1); the receive side stays clean.
- `smaps` of the leaking rank-0 worker shows the growth in ~18 large (>16 MB) anonymous regions (glibc heap arenas) — i.e. many small per-message `malloc`s that are never freed, not mmap'd USM regions.
- Not mitigated by `CCL_STAGING_BUFFER=regular`, `FI_MR_CACHE_MAX_COUNT=0`, or `FI_MR_CACHE_MONITOR=disabled` (leak identical: +936 MB).
- Under sustained 16-stream load the leak reaches ~1.1 GB/min, exhausting host RAM within hours (we hit full-host OOM twice before isolating this).
## Expected behaviour
Host memory of the PP workers stays bounded across requests, as it does for TP=2.
## Workaround
Switching the same deployment to `--tensor-parallel-size 2` eliminates the leak completely (verified under 16-stream production load: 12.59 → 12.60 GiB after 48 requests). Throughput cost on this MoE at 16 concurrent streams: 297 → 213 output tok/s, which we accept for an always-on service. A fix for the PP path would still be very welcome, since PP=2 is otherwise the fastest multi-user configuration on this no-P2P topology.
Happy to provide raw measurement scripts/logs or run instrumented builds.
Contributor guide
Assessment
This issue has not been assessed yet.