stackabletech / stackabletech/secret-operator
pods get oom-killed when multiple heavy spark applications start at the same time
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 13
- Forks
- 8
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 10
Description
Affected Stackable version
25.3.0
Current and expected behavior
when multiple spark applications get started at the same time, all of them having 300 executors, a lot of pvcs are submitted which need to be satisfied by the secret-operator. in basic configuration the secret operator container has 128mb of memory which seems to be not enough because in that case all the pods get oom-killed
Possible solution
Increase the memory limit of the secret-operator container from 128mb to 1gb
Additional context
@soenkeliebau like mentioned today, I'll try to get further details. I think this is reproducible by submitting a 3000 exec application and keeping an eye on the secret-operator ds
Environment
No response
Would you like to work on fixing this bug?
None
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the secret-operator DaemonSet in the basic configuration and reproduce the issue by submitting a Spark application with 3,000 executors while observing the secret-operator pods. Verify whether the 128 MB memory limit causes OOM kills, then confirm that the selected higher limit prevents them during concurrent heavy applications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust, spark
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100