dmlc / dmlc/xgboost

GPU memory usage when only using CPU.

Open
#12,116 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Even with `device=cpu`, 400 MiB is always allocated to GPU memory. It's allocated after calling `XGBGetGlobalConfig` c_api at https://github.com/dmlc/xgboost/blob/master/python-package/xgboost/config.py#L149.

It's a small portion, but it can be massive when using multi-processing, since each process allocates 400MiB on GPU. Especially when training ensembles, it's more efficient to train each ensemble concurrently across different processes, with 1 CPU per process.

The solution is to set the env var `CUDA_VISIBLE_DEVICES=""` or install `xgboost-cpu`.

I wonder if this can be solved natively in c_api.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at python-package/xgboost/config.py around line 149 and trace the XGBGetGlobalConfig C API call that follows device=cpu initialization. Reproduce the process with CPU-only settings and measure GPU memory, comparing the CUDA_VISIBLE_DEVICES="" and xgboost-cpu workarounds. Done means the native C API path no longer allocates the reported GPU memory for CPU-only use, with regression coverage for the configuration behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.