stackabletech / stackabletech/spark-k8s-operator

Feature request: Passing Tolerations to executor pods

Open
#240 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

customer-request
Dominant language
Rust
Stars
72
Forks
4
Avg merge
2d 15h
Merged PRs (30d)
15

Description

Affected version

23.4.0

Current and expected behavior

We are deploying Stackable on Azure with AKS using Helm/Terraform. We have successfully run SparkApplications on the default node pool. However, we would like to be able to deploy executors in a second node pool containing only Spot instances.
In Azure, all Spot instance node pools automatically get the taint kubernetes.azure.com/scalesetpriority=spot:NoSchedule (even if we do not specify it in the Terraform file, this taint is apparently mandatory).
Now, I can specify nodeAffinity to match the spot instances' labels, but I haven't found a way to pass tolerations. The helm chart for the Spark operator has a "tolerations" variable and I tried passing the right toleration there (as specified here), but it had no effect:
The executors will not schedule, since their affinity does not match the default node pool and they have no toleration for the spot

Is there a way to pass tolerations in a SparkApplication that I have just overlooked? If not: I think this would be a fairly relevant feature for pod placement. Are there any plans to implement this?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how SparkApplication executor pods are configured and compare that path with the Spark operator Helm chart's tolerations setting. Confirm how a toleration for the Azure Spot taint should be supplied in a SparkApplication and verify that executors can schedule on the Spot node pool without changing the existing node affinity behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, helm, kubernetes, rust, spark, terraform
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.