gchq / gchq/sleeper

Design options for managing persistent EMR clusters

Open
#7,453 0 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want a persistent EMR cluster to always be available, so that bulk import jobs will continue to run even if one fails.

As a user of Sleeper, I want to be able to deploy multiple clusters, e.g. one per availability zone, so that I can have redundancy and fallback if an availability zone fails.

### Description / Background

We've seen EMR clusters fail and disappear when they were created by the CDK. This suggests we need some alternative way to manage them that can ensure durability.

We'd like to investigate options for how we could create and manage persistent EMR clusters without the CDK. We'd like to document these either on this issue, or in a design document in the Git repository.

### Technical Notes / Implementation Details

We can consider:
- When we should create EMR clusters
- Whether and how we should track how many clusters the user wants in which subnets
- What should happen if a bulk import job is submitted and there are no available clusters
- What should happen when a cluster fails
- How this relates to potential improvements to retry failed bulk import jobs, e.g. with StepFunctions

It would be nice to avoid storing the state of the clusters in a way that duplicates EMR's own tracking, since that should be the source of truth.

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or entry points, so begin by surveying the existing CDK EMR cluster setup and bulk import job flow. Document options for persistent and multiple clusters, handling unavailable or failed clusters, retry integration, and using EMR as the source of truth for cluster state.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.