xgboost4j-spark-gpu fails to run on multi-GPU server with GPUs in exclusive process mode and spark-rapids plugin
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Exceptions result when running as above for both driver and executor processes. Stack traces for both includes
`
/workspace/src/common/cuda_rt_utils.cc: 60: cudaErrorDevicesUnavailable: CUDA-capable device(s) is/are busy or unavailable
`
indicating XGBoost code is attempting to create CUDA contexts even though the corresponding Spark driver and executor processes should either not be using GPUs (Spark driver) or have been assigned other device ids by Spark.
On the driver side, it seems one source (might be more than one with the first one triggering the error) might be coming from RabitTracker initialization which uses the common [InitNewThread function](https://github.com/dmlc/xgboost/blob/cd7f9974e996a5b30e9cd3530510a5ae291c6bbf/src/global_config.cc#L17) whose logic seems to setCurrentDevice on the default device. The source on the executor side is not clear at this point, but it is XGBoost code.
When switching GPUs to default process mode, I believe this behavior (depending on root cause above for executors, i.e. is it coming from main executor thread or new thread) can lead to race conditions in current stream/thread device settings with spark-rapids, resulting in spurious cuda kernels launched by executors on non-assigned device 0, in turn causing errors like:
```
Caused by: ai.rapids.cudf.CudfException: after determining tmp storage requirements for exclusive_scan: cudaErrorInvalidDevice: invalid device ordinal
```
first noted in https://github.com/NVIDIA/spark-rapids-examples/issues/565
Specifically, executor processes launching kernels on assigned device and device 0 can be observed in nsys traces.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with src/common/cuda_rt_utils.cc around line 60 and src/global_config.cc at InitNewThread, then trace RabitTracker initialization in xgboost4j-spark-gpu. Reproduce with Spark-RAPIDS on a multi-GPU server in exclusive process mode and compare driver and executor behavior, using nsys traces if available. Done means the unwanted CUDA contexts and device-0 kernel launches are identified and the reported failures no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, spark
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100