[FEA] Improve diagnostics for CUDA initialization failures
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
cuDF Java can report a misleading error when its first CUDA call fails:
```
Caused by: ai.rapids.cudf.CudaFatalException: Fatal CUDA error encountered at: ../../src/main/native/src/CudaJni.cpp:56: 35 cudaErrorInsufficientDriver CUDA driver version is insufficient for CUDA runtime version
at ai.rapids.cudf.Cuda.memGetInfo(Native Method)
```
`cudaErrorInsufficientDriver` can mean that:
* no NVIDIA driver or GPU is present;
* no GPU was assigned to the container;
* `libcuda.so.1` is not mounted or discoverable; or
* the host driver is too old for the bundled CUDA runtime.
The current message suggests only the last cause. In Kubernetes, this can lead
operators to change the cuDF artifact when GPU provisioning is the real issue.
**Describe the solution you'd like**
For `cudaErrorInsufficientDriver`, append a non-throwing, best-effort diagnostic
to the original exception. Do not invoke external tools such as `nvidia-smi`.
The probe could run once per process and report:
* the CUDA runtime version;
* whether `libcuda.so.1` can be loaded;
* if available, driver initialization status, driver-supported CUDA version,
and visible device count.
The message should distinguish among an unavailable driver library, no visible
GPU, and a driver/runtime version mismatch. If the probe is inconclusive, list
these possible causes instead of stating that the driver is necessarily old.
**Describe alternatives you've considered**
Applications can run platform-specific checks, but `nvidia-smi` may not be on
`PATH` and device-file checks are not portable.
`cudaDriverGetVersion()` is useful, but CUDA documents that it may itself return
initialization errors. Checking whether `libcuda.so.1` can be loaded provides
additional information without depending on an external executable.
At minimum, error 35 could list the likely causes even if cuDF cannot identify
which one occurred.
**Additional context**
The example originates from
[`CUDF_CUDA_TRY(cudaMemGetInfo(...))`](https://github.com/NVIDIA/cudf/blob/89dac81156890365ac3a28cbdfaa6b18f15f46ac/java/src/main/native/src/CudaJni.cpp#L56).
CUDA version API documentation:
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART____VERSION.html
Diagnostics should refer to the host driver's supported CUDA version, not the
host CUDA toolkit, because driver compatibility is the relevant boundary for a
bundled runtime.
Contributor guide
Research direction
Start at java/src/main/native/src/CudaJni.cpp:56, where CUDF_CUDA_TRY(cudaMemGetInfo(...)) produces the example, and review the CUDA version API documentation linked in the issue. Determine how the existing exception path can attach a best-effort diagnostic without external tools. Done means error 35 distinguishes unavailable libraries, missing visible GPUs, and driver/runtime mismatch, while preserving the original exception and handling inconclusive probes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- backend, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100