[GSD-13038] Regression in 26.09+: dual-GPU workloads eagerly pin a host-RAM mirror of every large device USM allocation on the peer device, exhausting system memory

## Summary

On a 2× Arc Pro B70 (Battlemage G31) system running PyTorch DDP (one rank per card,
xccl/oneCCL collectives), every large device-USM segment a rank allocates on its own
card is also registered on its file descriptor to the **peer** card as GTT (GPU-mapped
system memory). On 26.05 these peer registrations are bookkeeping only. On 26.09+
(observed on 26.22) they are **eagerly pinned to physical host pages**: ~17.7 GiB of
system RAM per rank for an ~18.5 GiB VRAM working set — 35.4 GiB total on a 64 GiB
machine. The kernel then hits order-0 page-allocation failures, the desktop swaps out,
and at production scale the workload dies. Downgrading only `intel-compute-runtime`
26.22 → 26.05 (nothing else changed) reduces pinned memory for the identical workload
from **25.9 GiB to 0.67 GiB** with identical throughput.

## Environment

| component | version |
|---|---|
| intel-compute-runtime | **26.22.38646.4 (bad)** / **26.05.37020.3 (good)** |
| GPUs | 2× Intel Arc Pro B70 (Battlemage G31, 8086:e223), 32 GiB VRAM each, PCIe (no XeLink) |
| kernel / driver | Linux 7.1.3 (CachyOS), `xe` |
| level-zero-loader | 1.28.6 |
| intel-graphics-compiler | 2.36.3 |
| intel-gmmlib | 22.10.0 |
| workload stack | PyTorch 2.13.0+xpu, oneCCL 2022.0 (xccl backend), torchrun 2 ranks |
| system RAM | 64 GiB |

## Reproducer

```python
# gtt_repro.py — torchrun --standalone --nproc-per-node=2 gtt_repro.py
import os, time
import torch
import torch.distributed as dist

rank = int(os.environ["RANK"])
local = int(os.environ["LOCAL_RANK"])
dev = torch.device(f"xpu:{local}")
dist.init_process_group("xccl", device_id=dev)

# 8 GiB of device USM on this rank's card, in 1 GiB segments
held = [torch.empty(1024 ** 3 // 4, dtype=torch.float32, device=dev) for _ in range(8)]
# gradient-bucket-sized collective traffic
bucket = torch.ones(25_000_000, device=dev)
for _ in range(50):
dist.all_reduce(bucket)
torch.xpu.synchronize(dev)

print(f"rank {rank} pid {os.getpid()} holding 8 GiB on {dev}; sleeping for inspection", flush=True)
time.sleep(120)
dist.barrier()
dist.destroy_process_group()
```

Inspect while it sleeps:

```sh
for pid in ; do
for fd in /proc/$pid/fdinfo/*; do grep -HE 'drm-resident-(gtt|vram0)' $fd; done
done
free -h # or a kernel with pinned-GPU-memory accounting in /proc/meminfo
```

Observed per-fd accounting for the reproducer (each rank holds 8 GiB on its own card;
note the ~8.9 GiB **GTT on the fd to the peer card**, mirror-imaged between ranks):

```
pid 116400 (rank 0, computes on renderD128)
fd4 /dev/dri/renderD129: drm-resident-gtt: 8913000 KiB drm-resident-vram0: 191268 KiB
fd5 /dev/dri/renderD128: drm-resident-gtt: 642792 KiB drm-resident-vram0: 9355696 KiB
pid 116401 (rank 1, computes on renderD129)
fd4 /dev/dri/renderD129: drm-resident-gtt: 638440 KiB drm-resident-vram0: 9348024 KiB
fd5 /dev/dri/renderD128: drm-resident-gtt: 9011816 KiB drm-resident-vram0: 392456 KiB
```

(The capture above is from 26.05, where the registrations exist but stay unpinned.
On 26.22 the same registrations are backed by pinned host pages — see below.)

## Bad vs good, identical real workload (PPO trainer, ~16 GiB VRAM working set per rank)

| | 26.22 | 26.05 |
|---|---|---|
| peer-fd `drm-resident-gtt` per rank | 12.8 GiB | 13.3 GiB (bookkeeping only) |
| pinned host memory, both ranks (kernel `GPUActive`) | **25.9 GiB** | **0.67 GiB** |
| throughput | ~3.0 s/wave | ~3.0 s/wave |

(`GPUActive` is the CachyOS kernel's pinned-GPU-page accounting in `/proc/meminfo`;
on a mainline kernel the same effect is visible as "used" memory in `free` that no
process RSS accounts for, disappearing when the workload exits.)

At full production scale on 26.22 the pinning reached 35.4 GiB (17.7 GiB per rank),
`free` showed ~7 GiB available of 64 GiB, the kernel logged order-0 allocation
failures in unrelated processes (`kwin_wayland: page allocation failure: order:0`),
and the training run died without OOM-killer involvement.

Growth is stepwise and exactly tracks the allocator's large segments: peer-fd GTT grew
in increments of 3,174,400 KiB — precisely the workload's 512×2048×795 fp32 buffer —
and plateaus at ~the VRAM working set once the caching allocator stops creating new
segments. Nothing in the workload allocates host or shared USM of this size; all
tensors are device-resident.

## What I ruled out

- **USM pooling knobs**: `NEOReadDebugKeys=1 EnableHostUsmAllocationPool=0
EnableDeviceUsmAllocationPool=0 EnableUsmAllocationPoolManager=0` on 26.22 → peer-fd
GTT unchanged; additionally the workload then crashes in backward with
`UR_RESULT_ERROR_OUT_OF_RESOURCES` (possibly its own issue).
- **Workload-side caching**: releasing unused allocator segments
(`torch.xpu.empty_cache()`) after warmup reclaims only the warmup-transient portion
(~2 GiB/rank); live segments remain mirrored and pinned.
- **VRAM oversubscription/eviction**: `drm-resident-vram0` ≈ `drm-total-vram0`
throughout; VRAM is never oversubscribed (≤ 19 of 32 GiB).

## Expected behavior

Cross-device registrations for peer access (if that is what these are) should not
eagerly pin physical host pages proportional to the peer's entire VRAM working set —
26.05's lazy behavior, which performs identically for this workload, demonstrates the
pinning is unnecessary.

## Possibly related

- **#952 looks like this same defect presenting as a crash at 4-GPU scale.** There:
4× B60, 60 GB RAM, ~11.4 GiB reserved per rank, `UR_RESULT_ERROR_OUT_OF_RESOURCES`
at the first large kernel — exactly what eager peer-pinning predicts (each rank's
working set mirrored per peer exceeds host RAM at 4 ranks; our 2-GPU/64 GiB system
survives at 35.4 GiB pinned but dies at the system level instead). Their
discriminating evidence (single-GPU OK, independent concurrent jobs OK, small-model
DDP OK, large-model DDP fails on 2/3/4 ranks) matches peer-mirror scaling, and their
USM-pool debug-key attempts not helping matches ours.
- #935: on these cards (BMG, cross-root-port) L0 dev-to-dev P2P is refused by the
kernel, so peer access is host-backed — presumably why these peer GTT registrations
exist at all. The regression here is only that 26.09+ pins them eagerly.
- The 25.40→26.09 window matches the `useUsmPoolManager` / "l0 device usm growing
pools" default flip referenced in #916 (also a dual-Arc, UR multi-device-context
setup), though disabling those pools via debug keys did not change the behavior
here, so the overlap may be coincidental.

Start by running gtt_repro.py with the stated two-rank torchrun command on compute-runtime 26.22 and inspect each rank's /proc//fdinfo entries while it sleeps. Compare peer-fd drm-resident-gtt and host-memory accounting with 26.05; done means peer registrations no longer eagerly pin host pages proportional to device-USM allocations, without reducing workload throughput.

Written by the indexing model from the issue text.