gchq / gchq/sleeper

Compare performance of EKS Auto Mode to EMR Serverless, Persistent EMR

Open
#7,540 0 comments 0 reactions 0 assignees View on GitHub
bulk-import-module
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Description / Background

Before we added the EKS Auto Mode integration, we had assumed that EMR Serverless would be the best option for bursty workloads, and Persistent EMR would be the best option for persistent workloads. It seems possible that EKS Auto Mode might be an improvement on both.

We'd like to consider whether the EMR implementations of bulk import are worth keeping when we have bulk import on EKS Auto Mode.

### Technical Notes / Implementation Details

One way to do this would be to take the existing EmrBulkImportPerformanceST, re-enable it on a branch, adapt it to run against both EMR Serverless and Persistent EMR, and compare the performance reported by that test to the performance we get in EksBulkImportPerformanceST.

We can also look at the delay between a job being submitted and starting in Spark, which currently is not measured in the performance test. We can compare this on a cold start versus an instance which has recently run a bulk import, because this will affect the delay if it needs to e.g. load a large Docker image. This seems likely to make a difference on EKS.

The outcome should be details of the testing done and the results, published on this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the existing EmrBulkImportPerformanceST and EksBulkImportPerformanceST performance tests. Re-enable and adapt the EMR test for EMR Serverless and Persistent EMR, then compare bulk-import performance and job-submission-to-Spark startup delay for cold and recently used instances. Done means the testing details and results are published on this issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, java, kubernetes, spark
Domain
cloud, data-engineering, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.