[Bug] CUDA caching allocator unable to satisfy 3-18 GiB contiguous allocation despite 50+ GiB free on RTX PRO 6000 Blackwell + WSL2 (16 GiB hidden CUDA overhead)

### 🐛 Describe the bug

PyTorch CUDA caching allocator on **NVIDIA RTX PRO 6000 Blackwell (sm_120, 96 GB VRAM)** under **WSL2 Ubuntu 22.04** reports `torch.OutOfMemoryError` for 3-18 GiB contiguous allocations even though 50+ GiB of GPU memory is free.

The error message itself is paradoxical:
```
torch.OutOfMemoryError: CUDA out of memory.
Tried to allocate 3.48 GiB.
GPU 0 has a total capacity of 95.59 GiB of which 50.40 GiB is free.
Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use.
Of the allocated memory 42.49 GiB is allocated by PyTorch,
and 47.72 MiB is reserved by PyTorch but unallocated.
```

Two issues here:

1. **Display bug**: `17179869184.00 GiB` should be `17179869184 bytes = 16 GiB`. Units are mislabeled in the error message — should be `~16 GiB` of "non-PyTorch memory".

2. **Allocation impossible despite free memory**: 50.40 GiB free on GPU, only 3.48 GiB requested → should easily fit. But PyTorch cannot satisfy it.

3. **`PYTORCH_ALLOC_CONF=expandable_segments:True` breaks CUDA driver entirely on Blackwell** with `RuntimeError: CUDA driver error: unknown error` — this hint in error message is actively harmful on this hardware.

## Versions

- PyTorch: tested **2.10.0+cu128** AND **2.11.0+cu130** — both show same behavior
- CUDA runtime: 12.8 (cu128) and 13.0 (cu130)
- NVIDIA driver: 596.36 (Windows host) and 581.80 RTX Enterprise — both same
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (sm_120, 97887 MiB)
- OS: Windows 11 LTSC 26100 / WSL2 2.6.3.0 / Ubuntu 22.04.5 LTS / kernel 6.6.87.2-microsoft-standard-WSL2

`collect_env.py` output: see attached `collect_env_output.txt`.

## Reproduction

Easiest reproducer is loading any Qwen3.5/3.6 hybrid Mamba model via vLLM (filed vllm#41619). Failing call site:

```python
# In vllm/v1/worker/gpu_model_runner.py:6537
tensor = torch.zeros(
[num_blocks, block_size, num_kv_heads, head_size],
dtype=torch.bfloat16,
device=self.device
)
# Tensor size: ~3.48 GiB contiguous
# OOM despite torch.cuda.mem_get_info() showing 50+ GiB free
```

Minimal script:
```python
import torch
torch.cuda.init()
free, total = torch.cuda.mem_get_info()
print(f"Free: {free/1024**3:.2f} GiB, Total: {total/1024**3:.2f} GiB")
# On Blackwell+WSL2 fresh init: free=~80 GiB, total=96 GiB
# But large alloc still fails!

# Allocate model-sized chunks first to fragment
chunks = []
for i in range(60): # 60 × 0.5 GiB = 30 GiB
chunks.append(torch.zeros(int(0.5 * 1024**3 / 4), dtype=torch.float32, device='cuda'))

# Now try big contiguous alloc:
big = torch.zeros(int(3.5 * 1024**3 / 4), dtype=torch.float32, device='cuda') # 3.5 GiB
# OOM here on Blackwell+WSL2 even though ~50 GiB still free
```

## Expected behavior

Allocator should either:
- (a) Find 3.48 GiB contiguous in 50+ GiB free space (defragment / coalesce free blocks)
- (b) Report **accurate** non-PyTorch memory in error message (16 GiB, not 17179869184 GiB)
- (c) Make `expandable_segments:True` work on sm_120 (currently breaks driver)

## Suggested fixes

1. **Fix display bug** in OOM error message — show `non-PyTorch memory: X GiB` correctly
2. **Coalesce free segments** before raising OOM (or at least try to)
3. **Investigate `expandable_segments` failure on sm_120** — currently raises `CUDA driver error: unknown error` instead of working
4. **Add a check** in allocator for `mem_get_info() free >> requested` and provide better diagnostic ("X GiB free but largest contiguous block is Y MiB")

## Why this matters

- Blocks all hybrid Mamba models (Qwen3.5/3.6 family) on Blackwell + WSL2
- Workstation Blackwell cards (RTX PRO 6000, 5000) are common AI dev hardware
- WSL2 is standard Windows AI dev environment
- Same code works on native Linux with same hardware (overhead there is 1-2 GiB, not 16 GiB)

## Related

- vllm-project/vllm#41619 (main bug report with full context)
- Possibly related to PyTorch CUDA allocator behavior under WSL2 GPU passthrough

## Files

- `collect_env_output.txt`
- `system-info.txt`
- `vllm-error-full.log` (real-world OOM trigger)

### Versions

Collecting environment information...
==============================
System Info
==============================
OS : Ubuntu 22.04.5 LTS (x86_64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version : Could not collect
CMake version : Could not collect
Libc version : glibc-2.35

==============================
PyTorch Info
==============================
PyTorch version : 2.10.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
XPU used to build PyTorch : N/A

==============================
Python Environment
==============================
Python version : 3.10.12 (main, Mar 3 2026, 11:56:32) [GCC 11.4.0] (64-bit runtime)
Python platform : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35

==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : Could not collect
CUDA_MODULE_LOADING set to :
GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Nvidia driver version : 596.36
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 14
On-line CPU(s) list: 0-13
Vendor ID: GenuineIntel
Model name: 13th Gen Intel(R) Core(TM) i5-13400F
CPU family: 6
Model: 191
Thread(s) per core: 2
Core(s) per socket: 7
Socket(s): 1
Stepping: 2
BogoMIPS: 4991.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 336 KiB (7 instances)
L1i cache: 224 KiB (7 instances)
L2 cache: 8.8 MiB (7 instances)
L3 cache: 20 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-13
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Vulnerable: No microcode
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cccl==13.2.75
[pip3] nvidia-cuda-crt==13.2.78
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvcc==13.2.78
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] nvidia-nvvm==13.2.78
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu128
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu128
[pip3] torchvision==0.25.0+cu128
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] Could not collect

==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.17.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

==============================
Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_nomugop

[collect_env_output.txt](https://github.com/user-attachments/files/27362418/collect_env_output.txt)
[system-info.txt](https://github.com/user-attachments/files/27362416/system-info.txt)
[vllm-error-full.log](https://github.com/user-attachments/files/27362417/vllm-error-full.log)

cc @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia