gchq / gchq/sleeper

Upgrade to EMR 7.13.0

Open
#7,455 0 comments 0 reactions 0 assignees View on GitHub
version-upgrades
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

See https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html

We'll need to match the versions of all the libraries listed there.

We'll need to adjust the version of the AWS SDK used for bulk import (`aws-java-sdk-v2.bulk-import.version` in the root pom.xml), to match the version in EMR. See the comment on that property in the pom.xml for how to find the right version.

We'll need to match the versions set for EMR in the instance property default values.

We'll need to update the base Spark Docker image used for bulk import on EKS.

In our root pom.xml, we manage the versions of a lot of the transitive dependencies of Spark. When we upgrade Spark to the new version, we'll need to go through every one of them to make sure we're not keeping back the version of a library for the sake of the old version of Spark.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.