Compare performance of EKS Auto Mode to EMR Serverless, Persistent EMR
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Description / Background
Before we added the EKS Auto Mode integration, we had assumed that EMR Serverless would be the best option for bursty workloads, and Persistent EMR would be the best option for persistent workloads. It seems possible that EKS Auto Mode might be an improvement on both.
We'd like to consider whether the EMR implementations of bulk import are worth keeping when we have bulk import on EKS Auto Mode.
### Technical Notes / Implementation Details
One way to do this would be to take the existing EmrBulkImportPerformanceST, re-enable it on a branch, adapt it to run against both EMR Serverless and Persistent EMR, and compare the performance reported by that test to the performance we get in EksBulkImportPerformanceST.
We can also look at the delay between a job being submitted and starting in Spark, which currently is not measured in the performance test. We can compare this on a cold start versus an instance which has recently run a bulk import, because this will affect the delay if it needs to e.g. load a large Docker image. This seems likely to make a difference on EKS.
The outcome should be details of the testing done and the results, published on this issue.
Contributor guide
Research direction
Start with the existing EmrBulkImportPerformanceST and EksBulkImportPerformanceST performance tests. Re-enable and adapt the EMR test for EMR Serverless and Persistent EMR, then compare bulk-import performance and job-submission-to-Spark startup delay for cold and recently used instances. Done means the testing details and results are published on this issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, java, kubernetes, spark
- Domain
- cloud, data-engineering, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100