dmlc / dmlc/xgboost

xgboost4j-spark-gpu fails to run on multi-GPU server with GPUs in exclusive process mode and spark-rapids plugin

Open
#11,884 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Exceptions result when running as above for both driver and executor processes. Stack traces for both includes
`
/workspace/src/common/cuda_rt_utils.cc: 60: cudaErrorDevicesUnavailable: CUDA-capable device(s) is/are busy or unavailable
`
indicating XGBoost code is attempting to create CUDA contexts even though the corresponding Spark driver and executor processes should either not be using GPUs (Spark driver) or have been assigned other device ids by Spark.

On the driver side, it seems one source (might be more than one with the first one triggering the error) might be coming from RabitTracker initialization which uses the common [InitNewThread function](https://github.com/dmlc/xgboost/blob/cd7f9974e996a5b30e9cd3530510a5ae291c6bbf/src/global_config.cc#L17) whose logic seems to setCurrentDevice on the default device. The source on the executor side is not clear at this point, but it is XGBoost code.

When switching GPUs to default process mode, I believe this behavior (depending on root cause above for executors, i.e. is it coming from main executor thread or new thread) can lead to race conditions in current stream/thread device settings with spark-rapids, resulting in spurious cuda kernels launched by executors on non-assigned device 0, in turn causing errors like:
```
Caused by: ai.rapids.cudf.CudfException: after determining tmp storage requirements for exclusive_scan: cudaErrorInvalidDevice: invalid device ordinal
```
first noted in https://github.com/NVIDIA/spark-rapids-examples/issues/565

Specifically, executor processes launching kernels on assigned device and device 0 can be observed in nsys traces.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/common/cuda_rt_utils.cc around line 60 and src/global_config.cc at InitNewThread, then trace RabitTracker initialization in xgboost4j-spark-gpu. Reproduce with Spark-RAPIDS on a multi-GPU server in exclusive process mode and compare driver and executor behavior, using nsys traces if available. Done means the unwanted CUDA contexts and device-0 kernel launches are identified and the reported failures no longer occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.