Design options for managing persistent EMR clusters
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a user of Sleeper, I want a persistent EMR cluster to always be available, so that bulk import jobs will continue to run even if one fails.
As a user of Sleeper, I want to be able to deploy multiple clusters, e.g. one per availability zone, so that I can have redundancy and fallback if an availability zone fails.
### Description / Background
We've seen EMR clusters fail and disappear when they were created by the CDK. This suggests we need some alternative way to manage them that can ensure durability.
We'd like to investigate options for how we could create and manage persistent EMR clusters without the CDK. We'd like to document these either on this issue, or in a design document in the Git repository.
### Technical Notes / Implementation Details
We can consider:
- When we should create EMR clusters
- Whether and how we should track how many clusters the user wants in which subnets
- What should happen if a bulk import job is submitted and there are no available clusters
- What should happen when a cluster fails
- How this relates to potential improvements to retry failed bulk import jobs, e.g. with StepFunctions
It would be nice to avoid storing the state of the clusters in a way that duplicates EMR's own tracking, since that should be the source of truth.
Contributor guide
Research direction
The issue names no repository files, tests, or entry points, so begin by surveying the existing CDK EMR cluster setup and bulk import job flow. Document options for persistent and multiple clusters, handling unavailable or failed clusters, retry integration, and using EMR as the source of truth for cluster state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100