dmlc / dmlc/xgboost

XGBoostAPIGuard calls cudaGetDevice() on every C API call

Open
#12,423 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Opening this as a tracking issue for the `XGBoostAPIGuard` / `cudaGetDevice()` overhead diagnosed in #12420, so it doesn't get lost — in that thread @trivialfis mentioned wanting to find a way to disable the CUDA call.

## Problem

Every C API call goes through `XGBoostAPIGuard`, which calls `cudaGetDevice()`. On a machine **without an NVIDIA driver** (mine is AMD-only), the CUDA 13 runtime shipped with the 3.4.0 Windows wheel appears to re-probe for the driver on every call instead of caching the failure. Each probe costs ~140µs of kernel time, and the probe path serializes across processes machine-wide.

Consequences:

- Single process: any workload making frequent C API calls (e.g. custom objectives, which make ~28 guarded calls per boosting round) pays a
constant per-call tax. In my case: per-fit kernel time went 0.03s → 0.77s, user time unchanged.
- Multi-process: because the probe serializes system-wide, the standard one-process-per-core pattern collapses. 16 workers on a 16C/32T machine:
~18 fits/s on 3.3.0 → ~1 fit/s on the 3.4.0 CUDA wheel, at ~6% CPU utilization and ~560k context switches/s.

## Minimal repro

```python
import time, numpy as np, xgboost
d = xgboost.DMatrix(np.zeros((10, 2)))
t0 = time.perf_counter()
for _ in range(2000):
d.num_row()
print(f"{(time.perf_counter() - t0) / 2000 * 1e6:.1f} us/call")
```

Results on the same machine (Windows 11, Ryzen 9 9950X, no NVIDIA driver):

| wheel | µs per call |
|---|---|
| xgboost 3.3.0 | 0.23 |
| xgboost 3.4.0 (CUDA) | **139** |
| xgboost-cpu 3.4.0 | 0.22 |

Note this doesn't reproduce on machines that have an NVIDIA driver installed.

## Environment

- xgboost 3.4.0 pip wheel (CUDA 13 build) vs 3.3.0 and xgboost-cpu 3.4.0
- Windows 11 Pro 10.0.26200
- AMD Ryzen 9 9950X (16C/32T), 64 GB RAM, no NVIDIA GPU/driver
- Python 3.14.6, numpy 2.5.1

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how XGBoostAPIGuard invokes cudaGetDevice() for each C API call, then run the minimal Python repro with repeated d.num_row() calls on a machine without an NVIDIA driver. Done means avoiding the repeated probe overhead while preserving the expected behavior for CUDA-enabled environments.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.