kubeflow / kubeflow/spark-operator

Support GPU resources in the SparkConnect CRD

Open
#2,958 6 comments 1 reaction 2 assignees Claimed by @Rajadi16 View on GitHub
good first issue help wanted kind/feature
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What feature you would like to be added?

Add GPU resource support to the SparkConnect API so the Connect server and (primarily) executor pods can request GPUs and have Spark's resource scheduler configured to use them.

### Why is this needed?

Kubeflow's primary audience runs ML workloads, and GPU-accelerated Spark could further enable that workflow. The NVIDIA RAPIDS Accelerator for Apache Spark offloads SQL/DataFrame execution to GPUs on executors, and GPU ML inference/training via pandas/Python UDFs is common for feature engineering and batch scoring. Spark Connect is a natural fit for these as interactive or long-lived sessions - exactly what the Kubeflow Spark client provides.

### Describe the solution you would like

_No response_

### Describe alternatives you have considered

_No response_

### Additional context

Relates to https://github.com/kubeflow/sdk/issues/466 and https://github.com/kubeflow/sdk/issues/470.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.