gchq / gchq/sleeper

Consider removing non-persistent EMR bulk import

Open
#7,399 0 comments 0 reactions 0 assignees View on GitHub
bulk-import-module
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a developer of Sleeper, I want to avoid maintaining systems that are not used, so that I can spend my time elsewhere.

### Description / Background

EmrBulkImportStack is an optional stack that runs bulk import on a separate EMR cluster for each job, which is then deleted after the job finishes.

This seems to be obsolete with the addition of EmrServerlessBulkImportStack. We'd like to consider removing it from Sleeper.

### Technical Notes / Implementation Details

This is in EmrBulkImportStack, EmrPlatformExecutor. There are a number of instance properties specific to it as well, and some system tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.