awslabs / awslabs/data-on-eks

[Feature] EMR on EKS with Volcano Scheduler

Open
#12 2 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Shell
Stars
857
Forks
303
Avg merge
11h 5m
Merged PRs (30d)
3

Description

**Part1 of the PR**
----------------
1/ Add this add-on deployment to internal TF modules . Here https://github.com/awslabs/data-on-eks/tree/main/workshop/modules/terraform-aws-eks-data-addons

2/ Add this add-on to [emr-eks-karpenter](https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/emr-eks-karpenter) pattern with a `create_volcano` variable and set it to `false` as default. Users will enable either Volcano or YuniKorn but not both

3/ Add an example under https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/emr-eks-karpenter/examples/nvme-ssd to show Volcano with gang scheduling

4/ Update the Website Docs to explain the execution process and the results

**Part2 of the PR**
----------------
1/ Add this add-on to [spark-k8s-operator](https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/spark-k8s-operator) pattern with a `create_volcano` variable and set it to `false` as default. Users will enable either Volcano or YuniKorn but not both

2/ Add an example under https://github.com/awslabs/data-on-eks/tree/main/analytics/terraform/spark-k8s-operator/examples/karpenter to show Volcano with gang scheduling

3/ Update the Website Docs to explain the execution process and the results

- New EMR on EKS deployment pattern with custom scheduler - [Volcano](https://volcano.sh/en/)
- Add an option add on

Use this example as a template and build on top of that.
https://github.com/awslabs/data-on-eks/tree/main/analytics/emr-eks-amp-amg

Contributor guide

Open the contributing guide

Research direction

Start by comparing analytics/emr-eks-amp-amg with the workshop/modules/terraform-aws-eks-data-addons module and the emr-eks-karpenter and spark-k8s-operator patterns. Review their examples directories and existing website documentation before running the relevant Terraform examples. Done means both patterns support an opt-in Volcano scheduler, each named example demonstrates gang scheduling, and the execution process and results are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, spark, terraform
Domain
cloud, data-engineering, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.