stackabletech / stackabletech/spark-k8s-operator
Make the History Server storage configuration shareable/reusable/discoverable
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 72
- Forks
- 4
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 15
Description
Spark jobs should be able to discover the location to store logs in the Spark History Server. Configuring this for each job is fragile and likely to lead to logs not being persisted after the job has finished. Jobs should be able to discover the location, access keys etc. for log storage.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the operator configures the Spark History Server and how Spark jobs currently receive log-storage settings. Define how the storage location and access keys can be exposed for reuse and discovery, then verify that jobs can persist logs after completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust, spark
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100