apache / apache/arrow

[CI][GPU] Add jobs for CUDA 13

Open
#48,783 3 comments 0 reactions 0 assignees View on GitHub
Component: Continuous Integration Component: GPU Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

The runs-on GPU instances we use are built [with CUDA 12](https://github.com/runs-on/runner-images-for-aws?tab=readme-ov-file#gpu). See log from `nvidia-smi`:
```
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
```

I tried installing `cuda-compat-13-0` as suggested [here](https://github.com/apache/arrow/pull/48583#issuecomment-3719253654):
```
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 13.0 |
|-----------------------------------------+------------------------+----------------------+
```

But the job still failed with:
```
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_IMAGE: This indicates that the device kernel image is invalid. This can also indicate an invalid CUDA module.
```

We might need to, build our own instances with packer installing newer versions of CUDA as suggested [here](https://runs-on.com/runners/gpu/#workflow-job-definition-1), wait until runs-on provides images with CUDA 13 or try reinstalling the drivers on the instance.

This is a follow up issue to:
- https://github.com/apache/arrow/issues/48582

### Component(s)

Continuous Integration, GPU

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.