KEP-107: SparkClient SDK integration with Kubeflow components
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What you would like to be added?
Reusable patterns, examples, and potentially KFP components that demonstrate how to use SparkClient alongside other Kubeflow components. The goal is to cover the primary ways data engineers and ML engineers would use Spark in a Kubeflow environment.
Potential use cases:
* SparkClient as a KFP pipeline step for data enrichment
* SparkClient + TrainerClient in a combined KFP pipeline
* Interactive Spark SQL in a Kubeflow Notebook
* Batch job submission in a KFP pipeline step
* Shared SparkConnect server across pipeline steps
### Why is this needed?
The SparkClient API is functional for standalone usage, but there are no examples or reusable components showing how to integrate it with the broader Kubeflow ecosystem. Concretely:
* No KFP component exists for SparkClient
* The Spark → Training workflow is a first-class use case with no working example
* Notebook usage is undocumented
* Batch job submission has no pipeline integration.
* Session sharing across pipeline steps is an untested pattern
### Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Research direction
Start by reading the SparkClient API and the existing KFP pipeline and TrainerClient entry points; the issue does not name specific files or tests. Compare the five proposed integration patterns and clarify which examples or reusable components are in scope. Done means the selected SparkClient workflows are demonstrated in Kubeflow, including any agreed notebook, batch, training, or session-sharing coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100