dmlc / dmlc/xgboost

Crash on Windows when `CUDA_VISIBLE_DEVICES` is set to `-1`

Open
#11,283 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

In a project where we combine XGBoost with Tensorflow within the same process, we ran into the following issue:

When the environment variable `CUDA_VISIBLE_DEVICES` is set to `-1`, the XGBoost predict step function crashes after about a minute of predicting. Strangely enough, it seems to happen stochastically. The crash only occurs after predicting for a while, either by setting `nthread` to a low value, or by repeating the same predict step many times. Doing the predict step once usually works without the crash, but not always.

The crash does not produce any error messages and only happens on Windows, as far as I can tell.

Here's a script to reproduce:

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

from rich.progress import track
import xgboost as xgb

def main():
features = xgb.DMatrix("features.buffer")
model_filenames = ["model_20210416_HCD2021_B.xgboost"]

prediction_list = []
for model_filename in track(model_filenames * 100):
xgb_model = xgb.Booster(model_file=model_filename)
prediction_list.append(xgb_model.predict(features))

print("Done without crashes!")

if __name__ == "__main__":
main()
```

Comment out the first two lines makes it work again.

pip freeze output:
```txt
numpy==2.0.2
scipy==1.13.1
xgboost==2.1.4
```
And with optional `rich` install for the progress bar (does not change the crash behavior):
```txt
markdown-it-py==3.0.0
mdurl==0.1.2
numpy==2.0.2
Pygments==2.19.1
rich==13.9.4
scipy==1.13.1
typing_extensions==4.12.2
xgboost==2.1.4
```

Files used: https://1drv.ms/u/c/cc884c602a30d109/ET6oclsK3PpLqnj6p4W0h40BU2vIMXQzQnOWRLl5SfecFw?e=eCoexz

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.