stackabletech / stackabletech/spark-k8s-operator
Feature request: Passing Tolerations to executor pods
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 72
- Forks
- 4
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 15
Description
Affected version
23.4.0
Current and expected behavior
We are deploying Stackable on Azure with AKS using Helm/Terraform. We have successfully run SparkApplications on the default node pool. However, we would like to be able to deploy executors in a second node pool containing only Spot instances.
In Azure, all Spot instance node pools automatically get the taint kubernetes.azure.com/scalesetpriority=spot:NoSchedule (even if we do not specify it in the Terraform file, this taint is apparently mandatory).
Now, I can specify nodeAffinity to match the spot instances' labels, but I haven't found a way to pass tolerations. The helm chart for the Spark operator has a "tolerations" variable and I tried passing the right toleration there (as specified here), but it had no effect:
The executors will not schedule, since their affinity does not match the default node pool and they have no toleration for the spot
Is there a way to pass tolerations in a SparkApplication that I have just overlooked? If not: I think this would be a fairly relevant feature for pod placement. Are there any plans to implement this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how SparkApplication executor pods are configured and compare that path with the Spark operator Helm chart's tolerations setting. Confirm how a toleration for the Azure Spot taint should be supplied in a SparkApplication and verify that executors can schedule on the Spot node pool without changing the existing node affinity behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, helm, kubernetes, rust, spark, terraform
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100