apache / apache/kyuubi

[TASK][MEDIUM] Integrate NVIDIA RAPIDS with Spark engine

Open
#5,466 4 comments 0 reactions 0 assignees View on GitHub
hacktoberfest
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before creating

- [X] I have searched in the [task list](https://github.com/orgs/apache/projects/296) and found no similar tasks.

### Mentor

- [X] I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

### Skill requirements

- Basic knowledge of Scala programming language, Kyuubi architecture, Apache Spark, NVIDIA RAPIDS
- Basic English writing skills

### Background and Goals

[RAPIDS](https://github.com/NVIDIA/spark-rapids) is a Spark plugin developed by NVIDIA to accelerate Apache Spark with GPUs.

As Kyuubi currently supports using Spark as the computing engine, we can add tests to integrate with RAPIDS in the Spark engine, and write docs to guide users in setting up Kyuubi with Spark plus RAPIDS to leverage GPUs.

### Implementation steps

- Add RAPIDS UTs in spark engine module, which could be conditionally enabled via maven profile and scalatest annotation
- Setup CI for RAPIDS testing (I'm not sure if GitHub Actions or ASF infra could provide such CUDA environments for running CI)
- Write docs to guide users in setting up Kyuubi with Spark plus RAPIDS to leverage GPUs.

### Additional context

Introduction of https://github.com/apache/kyuubi/issues/6232

Contributor guide

Open the contributing guide

Research direction

Start with the Spark engine module and inspect its existing unit tests, Maven profiles, and ScalaTest annotations. Determine how RAPIDS tests could be conditionally enabled and whether GitHub Actions or ASF infrastructure can provide CUDA environments. Done means RAPIDS integration tests, a workable CI approach, and user documentation for running Kyuubi with Spark and RAPIDS.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, distributed-systems, documentation, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.