XGBoostAPIGuard calls cudaGetDevice() on every C API call
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Opening this as a tracking issue for the `XGBoostAPIGuard` / `cudaGetDevice()` overhead diagnosed in #12420, so it doesn't get lost — in that thread @trivialfis mentioned wanting to find a way to disable the CUDA call.
## Problem
Every C API call goes through `XGBoostAPIGuard`, which calls `cudaGetDevice()`. On a machine **without an NVIDIA driver** (mine is AMD-only), the CUDA 13 runtime shipped with the 3.4.0 Windows wheel appears to re-probe for the driver on every call instead of caching the failure. Each probe costs ~140µs of kernel time, and the probe path serializes across processes machine-wide.
Consequences:
- Single process: any workload making frequent C API calls (e.g. custom objectives, which make ~28 guarded calls per boosting round) pays a
constant per-call tax. In my case: per-fit kernel time went 0.03s → 0.77s, user time unchanged.
- Multi-process: because the probe serializes system-wide, the standard one-process-per-core pattern collapses. 16 workers on a 16C/32T machine:
~18 fits/s on 3.3.0 → ~1 fit/s on the 3.4.0 CUDA wheel, at ~6% CPU utilization and ~560k context switches/s.
## Minimal repro
```python
import time, numpy as np, xgboost
d = xgboost.DMatrix(np.zeros((10, 2)))
t0 = time.perf_counter()
for _ in range(2000):
d.num_row()
print(f"{(time.perf_counter() - t0) / 2000 * 1e6:.1f} us/call")
```
Results on the same machine (Windows 11, Ryzen 9 9950X, no NVIDIA driver):
| wheel | µs per call |
|---|---|
| xgboost 3.3.0 | 0.23 |
| xgboost 3.4.0 (CUDA) | **139** |
| xgboost-cpu 3.4.0 | 0.22 |
Note this doesn't reproduce on machines that have an NVIDIA driver installed.
## Environment
- xgboost 3.4.0 pip wheel (CUDA 13 build) vs 3.3.0 and xgboost-cpu 3.4.0
- Windows 11 Pro 10.0.26200
- AMD Ryzen 9 9950X (16C/32T), 64 GB RAM, no NVIDIA GPU/driver
- Python 3.14.6, numpy 2.5.1
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how XGBoostAPIGuard invokes cudaGetDevice() for each C API call, then run the minimal Python repro with repeated d.num_row() calls on a machine without an NVIDIA driver. Done means avoiding the repeated probe overhead while preserving the expected behavior for CUDA-enabled environments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- api, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100