Yunikorn Scheduler not enabled in "Spark Operator with YuniKorn" module
- Dominant language
- Shell
- Stars
- 857
- Forks
- 303
- Avg merge
- 11h 5m
- Merged PRs (30d)
- 3
Description
## Description
Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration.
I deployed the solution as per instructions from the [Module Page](https://awslabs.github.io/data-on-eks/docs/blueprints/data-analytics/spark-operator-yunikorn)
However, it does not install Yunikorn nor it configrued the batch scheduler on Spark Operator. I had to manually update `enable_yunikorn` variable for TF and add `--enable-batch-scheduler=true & --kube-scheduler-names=yunikorn` flags to the operator
- [x] ✋ I have searched the open/closed issues and my issue is not listed.
## ⚠️ Note
Before you submit an issue, please perform the following for Terraform examples:
1. Remove the local `.terraform` directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): `rm -rf .terraform/`
2. Re-initialize the project root to pull down modules: `terraform init`
3. Re-attempt your terraform plan or apply and check if the issue still persists
## Versions
- Module version [Required]:
- Terraform version:
```
terraform -version
Terraform v1.5.7
on darwin_arm64
```
- Provider version(s):
```
+ provider registry.terraform.io/alekc/kubectl v2.1.3
+ provider registry.terraform.io/hashicorp/aws v5.98.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/helm v2.17.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.36.0
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/random v3.7.2
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0
```
## Reproduction Code [Required]
Steps to reproduce the behavior:
Just deploy the solution as is per the instruction from here : https://awslabs.github.io/data-on-eks/docs/blueprints/data-analytics/spark-operator-yunikorn
## Expected behavior
Should have installed Yunikorn and enabled it for Spark Operator
## Actual behavior
N/A
### Terminal Output Screenshot(s)
## Additional context
Contributor guide
Research direction
Start with the Spark Operator with YuniKorn module page and trace its Terraform and Helm configuration, including the enable_yunikorn variable and the Spark Operator scheduler flags mentioned in the report. Verify the deployment installs YuniKorn and enables the batch scheduler with yunikorn selected, then confirm the documented solution works without manual changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes, terraform
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100