kubeflow / kubeflow/spark-operator

Add a `Dependencies` field to `SparkConnectSpec` for structured JAR/package management

Open
#2,959 5 comments 1 reaction 0 assignees View on GitHub
good first issue help wanted kind/feature
Dominant language
Python
Stars
3.2k
Forks
1.5k
Avg merge
5d 10h
Merged PRs (30d)
13

Description

### What feature you would like to be added?

Add a `Dependencies` field to `SparkConnectSpec`, mirroring the existing `Dependencies` struct already present on `SparkApplicationSpec`.

### Why is this needed?

The Kubeflow SDK's `SparkClient` ([kubeflow/sdk#470](https://github.com/kubeflow/sdk/issues/470)) wants to offer first-class dependency management so that users can attach table-format and storage connectors (Iceberg, Delta Lake, Hudi, S3A) without hand-writing Spark config strings. Today, there is no structured way to declare dependencies in `SparkConnectSpec`.

`SparkApplication` already solves exactly this problem: its `Dependencies` struct is consumed by the submission path, which emits real `--jars`, `--packages`, `--exclude-packages`, and `--repositories` flags. `SparkConnect` is the only workload type in the operator without this capability, which is an inconsistency in the API surface and forces every higher-level tool to reinvent dependency handling on top of raw `sparkConf`.

### Describe the solution you would like

_No response_

### Describe alternatives you have considered

_No response_

### Additional context

Tracking issue on the SDK side: [kubeflow/sdk#470](https://github.com/kubeflow/sdk/issues/470).

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start by locating SparkConnectSpec and the existing Dependencies struct on SparkApplicationSpec, then trace the SparkConnect submission path. Confirm how dependencies become --jars, --packages, --exclude-packages, and --repositories flags. Done means SparkConnectSpec exposes equivalent structured dependency management and the submission path handles those fields consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python, spark
Domain
backend-api-design, distributed-systems, infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.