Test EKS bulk import on Graviton
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Description / Background
Under parent issue:
- https://github.com/gchq/sleeper/issues/7065
We'd like an option to run bulk import with EKS on ARM architecture.
We assume we'll want this to be the default, although it would be good to compare.
### Technical Notes / Implementation Details
Spark creates pods for the executors with its Kubernetes integration:
https://spark.apache.org/docs/3.5.6/running-on-kubernetes.html
Fargate on EKS doesn't support Graviton.
EKS Auto Mode lets you configure the architecture for a node pool:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks_v2-readme.html#eks-auto-mode
https://docs.aws.amazon.com/eks/latest/userguide/create-node-pool.html
#### Testing
We can compare the performance with a large bulk import job, e.g. 1 billion rows, between ARM and x86.
### Dependencies / Blockers
This seems likely to conflict with the following issue, as both work on the same CDK code:
- https://github.com/gchq/sleeper/issues/7070
It's possible that the following issue may also involve similar CDK changes:
- https://github.com/gchq/sleeper/issues/7066
The following issue implemented this on EKS Auto Mode, but didn't specifically test performance with Graviton:
- https://github.com/gchq/sleeper/issues/7668
Contributor guide
Research direction
Start with the CDK code related to EKS and the linked Spark Kubernetes integration and EKS Auto Mode documentation. Configure an ARM node pool for bulk import, then run a large job such as 1 billion rows on both Graviton and x86 and compare the performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, spark
- Domain
- cloud, data-engineering, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100