Check bulk import AWS SDK version against EMR Serverless image on a schedule
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a Sleeper maintainer, I want the bulk import AWS SDK version to be checked automatically against the
version EMR Serverless provides, so that I find out when the pin can be moved without having to inspect
the EMR Serverless Docker image by hand.
### Description / Background
`aws-java-sdk-v2.bulk-import.version` in `java/pom.xml` is pinned to whatever version of the AWS SDK v2 is
built into the EMR Serverless image, because the SDK classes from the image win on the classpath in the
Spark driver. If our version is too far ahead we get a mix of versions and failures like
`java.lang.NoSuchFieldError: AWS_AUTH_SCHEME_PREFERENCE` when `BulkImportJobDriver.start` creates an
SqsClient. See the comment on the property for the full background, and #8133 for the CI guard that stops
Dependabot bumping it.
The problem is that nothing tells us when the pin has gone stale. Dependabot raises a PR for this property
on every SDK release, `.github/scripts/checkVersionNumber.sh` fails it, and we close the PR - but that
signal says nothing about whether the pin could legitimately move. It only tells us the pin changed.
In practice it drifted a long way: the pin sat at 2.31.16 while EMR Serverless had moved on to 2.35.5
(the goodies jar went from 2.19.0 to 2.22.0), and we only found out because someone manually pulled the
image while triaging a Dependabot PR. That manual check is a docker pull of a multi-GB image plus some
unzipping, so nobody does it routinely.
We'd like a scheduled job that does that comparison for us and tells us when the two versions differ.
### Acceptance Criteria
**Given** the AWS SDK version in the EMR Serverless image matches `aws-java-sdk-v2.bulk-import.version`
**When** the scheduled check runs
**Then** it passes and does not notify anyone
**Given** the AWS SDK version in the EMR Serverless image differs from `aws-java-sdk-v2.bulk-import.version`
**When** the scheduled check runs
**Then** we are notified with both version numbers and the EMR release label they were read from
**Given** the EMR release label in the code has been changed
**When** the scheduled check runs
**Then** it reads the image for the new release label, without any further configuration
### Technical Notes / Implementation Details
Suggested shape: a new script `.github/scripts/checkBulkImportAwsSdkAgainstEmr.sh`, driven by a scheduled
workflow (weekly is probably enough - EMR releases are infrequent) that opens or updates a GitHub issue on
mismatch. Keep the existing fast check in `checkVersionNumber.sh` as the PR-time guard rather than making
PR builds pull a Docker image.
The commands below have all been run by hand against `emr-7.12.0` and produce the values quoted, but they
are a sketch rather than a finished script.
#### 1. Read the EMR release label from the code
`scripts/templates/instanceproperties.template` is generated by `GeneratePropertiesTemplates` and covered
by tests, so it stays in sync with the property defaults and is easier to parse than the Java source:
EMR_RELEASE=$(grep -oP '(?<=^# sleeper\.default\.table\.bulk\.import\.emr\.release\.label=).*' \
"${PROJECT_ROOT}/scripts/templates/instanceproperties.template")
# emr-7.12.0
This is the default of `DEFAULT_BULK_IMPORT_EMR_RELEASE_LABEL` in `NonPersistentEMRProperty`, which
`BULK_IMPORT_EMR_SERVERLESS_RELEASE` in `EMRServerlessProperty` inherits.
#### 2. Read the SDK version out of the EMR Serverless image
IMAGE="public.ecr.aws/emr-serverless/spark/${EMR_RELEASE}:latest"
docker pull --platform linux/amd64 "$IMAGE"
EMR_AWS_VERSION=$(docker run --rm --platform linux/amd64 --entrypoint /bin/bash "$IMAGE" -c \
'unzip -p /usr/share/aws/emr/serverless-goodies/lib/emr-serverless-spark-goodies-*.jar \
META-INF/maven/software.amazon.awssdk/sdk-core/pom.properties' \
| grep -oP '(?<=^version=).*')
# 2.35.5
All the SDK modules bundled in that jar share one version, so sdk-core is representative. This replaces
the decompiling of `VersionInfo` described in the older version of the pom comment.
To list the goodies jars and confirm the path when debugging:
docker run --rm --platform linux/amd64 --entrypoint /bin/bash "$IMAGE" -c \
'ls /usr/share/aws/emr/serverless-goodies/lib/'
The image is several GB. If disk space on the runner is a problem, the jar can be streamed out without
storing the image, using crane:
crane export --platform linux/amd64 "$IMAGE" - \
| tar -xO --wildcards 'usr/share/aws/emr/serverless-goodies/lib/emr-serverless-spark-goodies-*.jar' \
> goodies.jar
EMR_AWS_VERSION=$(unzip -p goodies.jar META-INF/maven/software.amazon.awssdk/sdk-core/pom.properties \
| grep -oP '(?<=^version=).*')
Remember to clean up afterwards either way:
docker rmi "$IMAGE"
#### 3. Read our pinned version and compare
cd "${PROJECT_ROOT}/java"
PINNED_AWS_VERSION=$(mvn help:evaluate -Dexpression=aws-java-sdk-v2.bulk-import.version -q -DforceStdout)
if [ "$PINNED_AWS_VERSION" != "$EMR_AWS_VERSION" ]; then
echo "Bulk import AWS SDK version is $PINNED_AWS_VERSION but $EMR_RELEASE provides $EMR_AWS_VERSION"
exit 1
fi
#### 4. Notify
From the workflow, on a non-zero exit, either fail the scheduled run or raise an issue, e.g.
gh issue create \
--title "Bulk import AWS SDK version can be updated to $EMR_AWS_VERSION" \
--body "$EMR_RELEASE provides $EMR_AWS_VERSION, we pin $PINNED_AWS_VERSION. \
Needs a run of EmrServerlessBulkImportST before merging."
Check for an existing open issue first (`gh issue list --search ...`) so a weekly schedule doesn't raise
a duplicate every week.
#### Things to watch out for
- The image must be pulled for `linux/amd64`. The architecture is also a Sleeper instance property
(`BULK_IMPORT_EMR_SERVERLESS_ARCHITECTURE`), currently constrained to X86_64, so this can stay fixed for
now but is worth a comment.
- Three places hold the version and must move together when it is bumped: the property in `java/pom.xml`,
the expected value in `.github/scripts/checkVersionNumber.sh`, and the goodies jar version quoted in the
pom comment. Worth considering whether this check can reduce that duplication.
- A mismatch is not by itself permission to bump the pin. Upgrading still needs a run of
`EmrServerlessBulkImportST` against a deployed instance, because the static version comparison doesn't
prove the mixed classpath works. The notification should say so.
- If we do automate raising the PR as well as detecting the drift, that should be a separate issue - the
test run makes it a poor fit for full automation.
### Dependencies / Blockers
Raised as a follow on ticket from: https://github.com/gchq/sleeper/pull/8175
Contributor guide
Research direction
Start with .github/scripts/checkVersionNumber.sh, java/pom.xml, and scripts/templates/instanceproperties.template, then inspect the GeneratePropertiesTemplates tests and the existing scheduled workflow conventions. Implement the proposed version comparison using the EMR Serverless image and ensure it reads the release label from the template. Done means a scheduled run compares both versions and opens or updates a mismatch issue without duplicate notifications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, github-actions, java, shell
- Domain
- ci-cd, cloud, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100