rapidsai / rapidsai/deployment
Databricks ML Runtime compatibility with CUDA 13
Open
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 15
- Forks
- 41
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 6
Description
As part of the testing for Databricks 18 LTS ML runtime after switching over the docs to use CUDA 13 in #719, there are two issues which surfaced that need further attention:
- Pandas 3.0 upgrade: RAPIDS 26.08 requires Pandas 3.0 at the minimum, and the version of pyspark installed does not fully support Pandas 3.0 yet which throws a bunch of warnings when trying to use Spark dataframes on Databricks. I did not notice any loss of functionality apart from the warnings, but definitely something to keep track of.
- pytorch CUDA 12 installed: The 18 ML LTS runtime comes with pytorch built against CUDA 12.9 installed. pytorch is considering dropping CUDA 12 support in future releases, so upgrading pytorch to CUDA 13 would ensure continued releases and better support with newer releases of RAPIDS.
CC: @ncclementi
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported behavior in the Databricks 18 LTS ML runtime, checking the installed Pandas, pyspark, PyTorch, and CUDA versions. Investigate the Pandas 3.0 warnings and the PyTorch CUDA 12.9 installation, then document or test an agreed resolution for both compatibility concerns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, pytorch
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100