awslabs / awslabs/data-on-eks

Yunikorn Scheduler not enabled in "Spark Operator with YuniKorn" module

Open
#839 1 comment 0 reactions 0 assignees View on GitHub
bug good first issue
Dominant language
Shell
Stars
857
Forks
303
Avg merge
11h 5m
Merged PRs (30d)
3

Description

## Description

Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration.
I deployed the solution as per instructions from the [Module Page](https://awslabs.github.io/data-on-eks/docs/blueprints/data-analytics/spark-operator-yunikorn)
However, it does not install Yunikorn nor it configrued the batch scheduler on Spark Operator. I had to manually update `enable_yunikorn` variable for TF and add `--enable-batch-scheduler=true & --kube-scheduler-names=yunikorn` flags to the operator

- [x] ✋ I have searched the open/closed issues and my issue is not listed.

## ⚠️ Note

Before you submit an issue, please perform the following for Terraform examples:

1. Remove the local `.terraform` directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): `rm -rf .terraform/`
2. Re-initialize the project root to pull down modules: `terraform init`
3. Re-attempt your terraform plan or apply and check if the issue still persists

## Versions

- Module version [Required]:

- Terraform version:
```
terraform -version
Terraform v1.5.7
on darwin_arm64
```

- Provider version(s):

```
+ provider registry.terraform.io/alekc/kubectl v2.1.3
+ provider registry.terraform.io/hashicorp/aws v5.98.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/helm v2.17.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.36.0
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/random v3.7.2
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0
```
## Reproduction Code [Required]

Steps to reproduce the behavior:
Just deploy the solution as is per the instruction from here : https://awslabs.github.io/data-on-eks/docs/blueprints/data-analytics/spark-operator-yunikorn

## Expected behavior
Should have installed Yunikorn and enabled it for Spark Operator

## Actual behavior
N/A

### Terminal Output Screenshot(s)

## Additional context

Contributor guide

Open the contributing guide

Research direction

Start with the Spark Operator with YuniKorn module page and trace its Terraform and Helm configuration, including the enable_yunikorn variable and the Spark Operator scheduler flags mentioned in the report. Verify the deployment installs YuniKorn and enables the batch scheduler with yunikorn selected, then confirm the documented solution works without manual changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes, terraform
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.