kubeflow / kubeflow/spark-operator
Add a `Dependencies` field to `SparkConnectSpec` for structured JAR/package management
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 1.5k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 13
Description
### What feature you would like to be added?
Add a `Dependencies` field to `SparkConnectSpec`, mirroring the existing `Dependencies` struct already present on `SparkApplicationSpec`.
### Why is this needed?
The Kubeflow SDK's `SparkClient` ([kubeflow/sdk#470](https://github.com/kubeflow/sdk/issues/470)) wants to offer first-class dependency management so that users can attach table-format and storage connectors (Iceberg, Delta Lake, Hudi, S3A) without hand-writing Spark config strings. Today, there is no structured way to declare dependencies in `SparkConnectSpec`.
`SparkApplication` already solves exactly this problem: its `Dependencies` struct is consumed by the submission path, which emits real `--jars`, `--packages`, `--exclude-packages`, and `--repositories` flags. `SparkConnect` is the only workload type in the operator without this capability, which is an inconsistency in the API surface and forces every higher-level tool to reinvent dependency handling on top of raw `sparkConf`.
### Describe the solution you would like
_No response_
### Describe alternatives you have considered
_No response_
### Additional context
Tracking issue on the SDK side: [kubeflow/sdk#470](https://github.com/kubeflow/sdk/issues/470).
### Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Research direction
Start by locating SparkConnectSpec and the existing Dependencies struct on SparkApplicationSpec, then trace the SparkConnect submission path. Confirm how dependencies become --jars, --packages, --exclude-packages, and --repositories flags. Done means SparkConnectSpec exposes equivalent structured dependency management and the submission path handles those fields consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python, spark
- Domain
- backend-api-design, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100