[EMR on EKS] Increase Job driver resiliency by running spark driver using Kubernetes Job controller
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
When you run spark job using EMR on EKS, it runs job-runner, driver and executors. EMR on EKS automatically retries up to 6 times if job-runner pod experiences failure. However, driver pod cannot be retried in case of failures. This feature is to run driver pod using Kubernetes Job controller so that customers can use `restartPolicy` and `backoffLimit` to improve the resiliency for driver pods. You can control if driver pod can be restarted using `restartPolicy = "OnFailure"` and specify amount of retries using `backoffLimit`.
**Which service(s) is this request for?**
EMR on EKS
Contributor guide
Research direction
The issue names EMR on EKS driver pods and the Kubernetes Job controller, but no repository files, tests, or entry points. Start by locating the driver and job-runner implementation, then verify how restartPolicy and backoffLimit should be exposed; done means driver failures can be retried through the Kubernetes Job controller.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100