[Bug]: Adjacent pods allocating the host-memory-offloading stalling other pods

System Info

Failure path 1 — node-wide CUDA/NVML stall during the pin window
During the ~12-minute pinned allocation of a ~700GB host KV cache, CUDA/NVML calls block node-wide:

All 4 of the neighbor's ranks sat stuck in torch.cuda.mem_get_info() — a plain CUDA API that blocks only on driver locks.
DCGM went blind on all 8 GPUs exactly for the pin window (resuming seconds after the pin completed), while node-exporter stayed fine — not a scrape gap.
Zero Xids; all node health checks green.
The healthy sibling therefore trips TRT-LLM's 300s hang detector → health POISONED → 503 → liveness kill → its restart's own pin window stalls the first worker — a self-perpetuating ping-pong.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Expected behavior

adjacent pod allocating 500GB of memory over 2min should not stall adjacent pods

actual behavior

additional notes

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start by reproducing the pinned host-KV allocation while observing torch.cuda.mem_get_info(), DCGM, node-exporter, and the TRT-LLM 300s hang detector on adjacent pods. Trace the allocation and CUDA/NVML interactions to establish why the node-wide stall occurs. Done means adjacent pods remain responsive during allocation and the reproduction no longer triggers health poisoning or restart ping-pong.