Review EKS vs EMR configuration properties
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Description / Background
Under parent issue:
- https://github.com/gchq/sleeper/issues/7065
Follows up from:
- https://github.com/gchq/sleeper/issues/7066
There was a review on the previous issue's PR after it was merged:
- https://github.com/gchq/sleeper/pull/7135
See the PR comments for changes.
### Technical Notes / Implementation Details
As well as the review comments, we can review which properties should be specific to EMR and EKS, and which should be shared.
A general rule can be anything to do with cores / memory / executors should be controllable independently for each type of bulk import, whereas things like shuffle parameters can probably be common. If we do make those independently settable that's probably okay too.
HadoopConfigurationProvider has two unused methods that can be removed, getConfigurationForEKS and getConfigurationForEMR.
#### Spark cores
We provision 5 cores per Spark executor, but that doesn't scale neatly. We could consider moving 4 core per executor on every bulk import mode.
Contributor guide
Assessment
This issue has not been assessed yet.