kubeflow / kubeflow/sdk

KEP-107: SparkClient SDK integration with Kubeflow components

Open
#471 2 comments 2 reactions 0 assignees View on GitHub
area/spark kind/feature needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What you would like to be added?

Reusable patterns, examples, and potentially KFP components that demonstrate how to use SparkClient alongside other Kubeflow components. The goal is to cover the primary ways data engineers and ML engineers would use Spark in a Kubeflow environment.

Potential use cases:
* SparkClient as a KFP pipeline step for data enrichment
* SparkClient + TrainerClient in a combined KFP pipeline
* Interactive Spark SQL in a Kubeflow Notebook
* Batch job submission in a KFP pipeline step
* Shared SparkConnect server across pipeline steps

### Why is this needed?

The SparkClient API is functional for standalone usage, but there are no examples or reusable components showing how to integrate it with the broader Kubeflow ecosystem. Concretely:
* No KFP component exists for SparkClient
* The Spark → Training workflow is a first-class use case with no working example
* Notebook usage is undocumented
* Batch job submission has no pipeline integration.
* Session sharing across pipeline steps is an untested pattern

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start by reading the SparkClient API and the existing KFP pipeline and TrainerClient entry points; the issue does not name specific files or tests. Compare the five proposed integration patterns and clarify which examples or reusable components are in scope. Done means the selected SparkClient workflows are demonstrated in Kubeflow, including any agreed notebook, batch, training, or session-sharing coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.