gchq / gchq/sleeper

Test EKS bulk import on Graviton

Open
#7,068 1 comment 0 reactions 0 assignees View on GitHub
cdk-module enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Description / Background

Under parent issue:
- https://github.com/gchq/sleeper/issues/7065

We'd like an option to run bulk import with EKS on ARM architecture.

We assume we'll want this to be the default, although it would be good to compare.

### Technical Notes / Implementation Details

Spark creates pods for the executors with its Kubernetes integration:

https://spark.apache.org/docs/3.5.6/running-on-kubernetes.html

Fargate on EKS doesn't support Graviton.

EKS Auto Mode lets you configure the architecture for a node pool:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks_v2-readme.html#eks-auto-mode
https://docs.aws.amazon.com/eks/latest/userguide/create-node-pool.html

#### Testing

We can compare the performance with a large bulk import job, e.g. 1 billion rows, between ARM and x86.

### Dependencies / Blockers

This seems likely to conflict with the following issue, as both work on the same CDK code:
- https://github.com/gchq/sleeper/issues/7070

It's possible that the following issue may also involve similar CDK changes:
- https://github.com/gchq/sleeper/issues/7066

The following issue implemented this on EKS Auto Mode, but didn't specifically test performance with Graviton:
- https://github.com/gchq/sleeper/issues/7668

Contributor guide

Open the contributing guide

Research direction

Start with the CDK code related to EKS and the linked Spark Kubernetes integration and EKS Auto Mode documentation. Configure an ARM node pool for bulk import, then run a large job such as 1 billion rows on both Graviton and x86 and compare the performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, spark
Domain
cloud, data-engineering, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.