GPU memory usage when only using CPU.
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Even with `device=cpu`, 400 MiB is always allocated to GPU memory. It's allocated after calling `XGBGetGlobalConfig` c_api at https://github.com/dmlc/xgboost/blob/master/python-package/xgboost/config.py#L149.
It's a small portion, but it can be massive when using multi-processing, since each process allocates 400MiB on GPU. Especially when training ensembles, it's more efficient to train each ensemble concurrently across different processes, with 1 CPU per process.
The solution is to set the env var `CUDA_VISIBLE_DEVICES=""` or install `xgboost-cpu`.
I wonder if this can be solved natively in c_api.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at python-package/xgboost/config.py around line 149 and trace the XGBGetGlobalConfig C API call that follows device=cpu initialization. Reproduce the process with CPU-only settings and measure GPU memory, comparing the CUDA_VISIBLE_DEVICES="" and xgboost-cpu workarounds. Done means the native C API path no longer allocates the reported GPU memory for CPU-only use, with regression coverage for the configuration behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100