stackabletech / stackabletech/spark-k8s-operator
Add additional Spark Properties via Configmap
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 72
- Forks
- 4
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 15
Description
As a user I want to be able to pass spark properties, which might be the same over multiple sparkApplications (e.g. kerberos settings), via a configmap to the spark-submit command. Using this, I save effort and complexity repeatedly defining the same spark properties over and over when writing sparkApplications.
I tried to do some hacks to mock this behavior with mounting a spark-defaults.conf (which won't work because the spark-driver already mounts on /opt/spark/conf and thus I can't mount it there because the location is not unique) or passing a --properties-file (which also does not work because a --properties-file is already passed inherently)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the operator builds the spark-submit command and handles SparkApplication configuration. Add a ConfigMap-based path for shared Spark properties without conflicting with the existing properties-file argument or /opt/spark/conf mount. Done means multiple SparkApplications can reuse the same properties, including Kerberos settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust, spark
- Domain
- data-engineering, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100