kubeflow / kubeflow/spark-operator
Support GPU resources in the SparkConnect CRD
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
### What feature you would like to be added?
Add GPU resource support to the SparkConnect API so the Connect server and (primarily) executor pods can request GPUs and have Spark's resource scheduler configured to use them.
### Why is this needed?
Kubeflow's primary audience runs ML workloads, and GPU-accelerated Spark could further enable that workflow. The NVIDIA RAPIDS Accelerator for Apache Spark offloads SQL/DataFrame execution to GPUs on executors, and GPU ML inference/training via pandas/Python UDFs is common for feature engineering and batch scoring. Spark Connect is a natural fit for these as interactive or long-lived sessions - exactly what the Kubeflow Spark client provides.
### Describe the solution you would like
_No response_
### Describe alternatives you have considered
_No response_
### Additional context
Relates to https://github.com/kubeflow/sdk/issues/466 and https://github.com/kubeflow/sdk/issues/470.
### Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Assessment
This issue has not been assessed yet.