CUDA OOM when training XGBoost with DaskQuantileDMatrix on high-dimensional dataset

## Problem Description

I'm trying to train XGBoost on a large, high-dimensional dataset using `DaskQuantileDMatrix` with 8 GPUs, but I'm encountering out-of-memory errors during training. According to [issue #11592](https://github.com/dmlc/xgboost/issues/11592), distributed training with `QuantileDMatrix` should allow training on large datasets by sharding data across workers. However, even with substantial GPU memory (8× H20 GPUs with 141GB each, totaling ~1.1TB), training fails.

## Environment

- **XGBoost version**: 3.0.2 (from rapids-25.12 conda channel)
- **RAPIDS version**: 25.12
- **Dask-CUDA version**: 25.12
- **Python version**: 3.11
- **CUDA version**: 12.x
- **GPU**: 8× NVIDIA H20 (141GB VRAM each, total 1128GB)
- **System RAM**: 1.96 TiB

## Dataset Information

- **Rows**: 53,844,756
- **Features**: 3,391
- **Total elements**: ~182.6 billion
- **Estimated raw size (float32)**: ~730 GB

## Minimal Reproducible Code

```python
import os
GPUs = ','.join([str(i) for i in range(0, 8)])
os.environ['CUDA_VISIBLE_DEVICES'] = GPUs

import dask_cudf
from dask.distributed import Client
from dask_cuda import LocalCUDACluster
import xgboost as xgb
from xgboost import dask as dxgb

# Setup cluster
cluster = LocalCUDACluster(
device_memory_limit='115GB',
jit_unspill=True,
protocol="ucx",
rmm_pool_size="136GB"
)
client = Client(cluster)

# Load data
data = dask_cudf.read_parquet('/path/to/data/*.pq').persist()

FEATURES = [f for f in data.columns if f != 'target']
print(f"Features: {len(FEATURES)}") # 3391

# Create DaskQuantileDMatrix
dtrain = dxgb.DaskQuantileDMatrix(client, data[FEATURES], data['target'], max_bin=128)

# Training parameters
xgb_params = {
"objective": "reg:absoluteerror",
"eval_metric": "mae",
"learning_rate": 0.1,
"max_depth": 10,
"max_bin": 128,
"subsample": 0.9,
"tree_method": "hist",
"device": "cuda"
}

# Train - OOM occurs here
output = xgb.dask.train(
client,
xgb_params,
dtrain,
num_boost_round=100,
evals=[(dtrain, "train")],
)
```

## Error Message

```
[error] [A][Stream 0x2][Upstream 54756745984B][FAILURE maximum pool size exceeded:
std::bad_alloc: out_of_memory: CUDA error (failed to allocate 54756745984 bytes)]

2025-12-15 02:36:03,562 - distributed.worker - WARNING - RMM allocation of 47.51 GiB failed,
spill-on-demand couldn't find any device memory to spill.

```

Each worker is attempting to allocate approximately **51-55 GB** of GPU memory but failing, even though:
- RMM pool is set to 136GB per GPU
- Device memory limit is 115GB
- H20 has 141GB total VRAM

## Analysis

Comparing with issue #11592:
| Metric | Issue #11592 | My case |
|--------|-------------|---------|
| Rows | 1.2 billion | 53.8 million |
| Features | 122 | **3,391** |
| Total elements | 146.4 billion | **182.6 billion** |
| GPUs | 6× H100 (80GB) | 8× H20 (141GB) |
| Total VRAM | 480 GB | 1,128 GB |
| Result | Success (~7 min) | **OOM** |

Despite having more than 2× the total GPU memory and fewer rows, training fails. The key difference is the **number of features (3391 vs 122)**.

## Main Question

**Is there any way to make this work, even if it's slower?**

I understand that training on such a large, high-dimensional dataset is challenging. I'm willing to accept slower training speeds if it means the training can complete successfully.

**Can `ExtMemQuantileDMatrix` help in this scenario?**
- According to the [external memory documentation](https://xgboost.readthedocs.io/en/stable/tutorials/external_memory.html), `ExtMemQuantileDMatrix` is designed to handle datasets larger than GPU memory by spilling to host memory or disk.
- Would this work with Dask distributed training? Or is it only for single-node scenarios?
- I noticed in issue #11592 that `ExtMemQuantileDMatrix` requires CUDA heterogeneous memory management or RMM support for efficient operation. My H20 GPUs have 141GB VRAM - would the large memory help offset the lack of NVLink/C2C?

## What I've Tried

- Reducing `max_bin` to 128 (from default 256)
- Configuring RMM pool (136GB) and device memory limit (115GB)
- Enabling `jit_unspill=True` for spill-to-host support
- Using UCX protocol for faster communication

## Expected Behavior

With 1,128 GB of total GPU memory distributed across 8 workers, I would expect to be able to train on a ~730 GB dataset (raw float32 size), especially when using `DaskQuantileDMatrix` which should compress the data through quantization.

Any guidance on how to handle high-dimensional datasets with distributed GPU training would be greatly appreciated!

Start by reproducing the supplied DaskQuantileDMatrix example with the stated XGBoost, RAPIDS, Dask-CUDA, and GPU configuration. Compare the allocation path with issue #11592 and the external-memory documentation; completion would require identifying a supported way to finish training or defining a reproducible fix for the high-dimensional allocation failure.

Written by the indexing model from the issue text.