gchq / gchq/sleeper

Review EKS vs EMR configuration properties

Open
#7,184 0 comments 0 reactions 0 assignees View on GitHub
bulk-import-module enhancement needs-refinement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Description / Background

Under parent issue:
- https://github.com/gchq/sleeper/issues/7065

Follows up from:
- https://github.com/gchq/sleeper/issues/7066

There was a review on the previous issue's PR after it was merged:
- https://github.com/gchq/sleeper/pull/7135

See the PR comments for changes.

### Technical Notes / Implementation Details

As well as the review comments, we can review which properties should be specific to EMR and EKS, and which should be shared.

A general rule can be anything to do with cores / memory / executors should be controllable independently for each type of bulk import, whereas things like shuffle parameters can probably be common. If we do make those independently settable that's probably okay too.

HadoopConfigurationProvider has two unused methods that can be removed, getConfigurationForEKS and getConfigurationForEMR.

#### Spark cores

We provision 5 cores per Spark executor, but that doesn't scale neatly. We could consider moving 4 core per executor on every bulk import mode.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.