gchq / gchq/sleeper

Persistent EMR cluster resilience

Open
#6,416 1 comment 0 reactions 0 assignees View on GitHub
enhancement parent-issue under-discussion
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want a persistent EMR cluster to always be available, so that bulk import jobs will continue to run even if one fails.

As a user of Sleeper, I want to be able to deploy multiple clusters, e.g. one per availability zone, so that I can have redundancy and fallback if an availability zone fails.

### Description / Background

Under epic:
- https://github.com/gchq/sleeper/issues/5078

We've seen persistent EMRs regularly fail in the middle of the night and never come back up. We've seen the CDK delete and recreate the cluster unexpectedly.

One potential improvement would be to deploy multiple persistent EMR clusters in an instance of Sleeper.

It would be nice to be able to create and remove clusters without redeploying the whole instance in the CDK.

This may be related to:
- https://github.com/gchq/sleeper/issues/6140

### Technical Notes / Implementation Details

To do next:
- https://github.com/gchq/sleeper/issues/7453

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.