[TASK][MEDIUM] Integrate NVIDIA RAPIDS with Spark engine
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before creating
- [X] I have searched in the [task list](https://github.com/orgs/apache/projects/296) and found no similar tasks.
### Mentor
- [X] I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.
### Skill requirements
- Basic knowledge of Scala programming language, Kyuubi architecture, Apache Spark, NVIDIA RAPIDS
- Basic English writing skills
### Background and Goals
[RAPIDS](https://github.com/NVIDIA/spark-rapids) is a Spark plugin developed by NVIDIA to accelerate Apache Spark with GPUs.
As Kyuubi currently supports using Spark as the computing engine, we can add tests to integrate with RAPIDS in the Spark engine, and write docs to guide users in setting up Kyuubi with Spark plus RAPIDS to leverage GPUs.
### Implementation steps
- Add RAPIDS UTs in spark engine module, which could be conditionally enabled via maven profile and scalatest annotation
- Setup CI for RAPIDS testing (I'm not sure if GitHub Actions or ASF infra could provide such CUDA environments for running CI)
- Write docs to guide users in setting up Kyuubi with Spark plus RAPIDS to leverage GPUs.
### Additional context
Introduction of https://github.com/apache/kyuubi/issues/6232
Contributor guide
Research direction
Start with the Spark engine module and inspect its existing unit tests, Maven profiles, and ScalaTest annotations. Determine how RAPIDS tests could be conditionally enabled and whether GitHub Actions or ASF infrastructure can provide CUDA environments. Done means RAPIDS integration tests, a workable CI approach, and user documentation for running Kyuubi with Spark and RAPIDS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, distributed-systems, documentation, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100