KEP-107: Support jar file submission for Spark jobs
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
### What you would like to be added?
The goal of the issue is to expose a `main_class` parameter on the `FileJob` dataclass to enable jar file submission.
### Why is this needed?
The Spark SDK KEP proposes a [`main_class`](https://github.com/kubeflow/sdk/blob/main/proposals/107-spark-client/README.md?plain=1#L325) parameter on the `FileJob` dataclass.
However, the current implementation for `FileJob` does not support this parameter. As such, jar file submission is not enabled on the Spark SDK.
### Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Research direction
Start with the FileJob dataclass implementation and compare it with the main_class parameter described in proposals/107-spark-client/README.md. Trace the existing Spark job submission entry point and check nearby tests for FileJob. Done means jar submissions can provide main_class without breaking existing file-job behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100