aws / aws/containers-roadmap

[EMR on EKS] Increase Job driver resiliency by running spark driver using Kubernetes Job controller

Open
#1,867 0 comments 3 reactions 0 assignees View on GitHub
EKS EMR Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
When you run spark job using EMR on EKS, it runs job-runner, driver and executors. EMR on EKS automatically retries up to 6 times if job-runner pod experiences failure. However, driver pod cannot be retried in case of failures. This feature is to run driver pod using Kubernetes Job controller so that customers can use `restartPolicy` and `backoffLimit` to improve the resiliency for driver pods. You can control if driver pod can be restarted using `restartPolicy = "OnFailure"` and specify amount of retries using `backoffLimit`.

**Which service(s) is this request for?**
EMR on EKS

Contributor guide

Open the contributing guide

Research direction

The issue names EMR on EKS driver pods and the Kubernetes Job controller, but no repository files, tests, or entry points. Start by locating the driver and job-runner implementation, then verify how restartPolicy and backoffLimit should be exposed; done means driver failures can be retried through the Kubernetes Job controller.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.