[Feature] EMR on EKS with Volcano Scheduler
- Dominant language
- Shell
- Stars
- 857
- Forks
- 303
- Avg merge
- 11h 5m
- Merged PRs (30d)
- 3
Description
**Part1 of the PR**
----------------
1/ Add this add-on deployment to internal TF modules . Here https://github.com/awslabs/data-on-eks/tree/main/workshop/modules/terraform-aws-eks-data-addons
2/ Add this add-on to [emr-eks-karpenter](https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/emr-eks-karpenter) pattern with a `create_volcano` variable and set it to `false` as default. Users will enable either Volcano or YuniKorn but not both
3/ Add an example under https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/emr-eks-karpenter/examples/nvme-ssd to show Volcano with gang scheduling
4/ Update the Website Docs to explain the execution process and the results
**Part2 of the PR**
----------------
1/ Add this add-on to [spark-k8s-operator](https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/spark-k8s-operator) pattern with a `create_volcano` variable and set it to `false` as default. Users will enable either Volcano or YuniKorn but not both
2/ Add an example under https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/spark-k8s-operator/examples/karpenter to show Volcano with gang scheduling
3/ Update the Website Docs to explain the execution process and the results
- New EMR on EKS deployment pattern with custom scheduler - [Volcano](https://volcano.sh/en/)
- Add an option add on
Use this example as a template and build on top of that.
https://github.com/awslabs/data-on-eks/tree/main/analytics/emr-eks-amp-amg
Contributor guide
Research direction
Start by comparing analytics/emr-eks-amp-amg with the workshop/modules/terraform-aws-eks-data-addons module and the emr-eks-karpenter and spark-k8s-operator patterns. Review their examples directories and existing website documentation before running the relevant Terraform examples. Done means both patterns support an opt-in Volcano scheduler, each named example demonstrates gang scheduling, and the execution process and results are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, spark, terraform
- Domain
- cloud, data-engineering, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100