Split EKS Spark jobs across subnets/AZs
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a user of Sleeper running bulk import on EKS, I want different bulk import jobs to be spread across availability zones, so that I can take advantage of capacity in all availability zones, and if one availability zone is unavailable jobs will still be processed.
### Description / Background
Under parent issue:
- https://github.com/gchq/sleeper/issues/7065
Currently in EksBulkImportStack we choose one subnet to run bulk import jobs in, and they all run there. A subnet is in a single availability zone, so that means all bulk import jobs will run in one availability zone.
We'd like to run jobs across all subnets.
### Technical Notes / Implementation Details
Currently EksBulkImportStack creates one Fargate profile, associated with one Kubernetes namespace.
We could create a separate Fargate profile and Kubernetes namespace for each subnet, and submit bulk import jobs to all of them.
There are two places where we configure which Kubernetes namespace is used. For the pods created by Spark, we set `spark.kubernetes.namespace` in StateMachinePlatformExecutor. For the initial Kubernetes job to submit to Spark, the namespace is set in the StepFunctions definition in `java/deployment/cdk/src/main/resources/step-functions/run-job.json`. That gets used in EksBulkImportStack.
We'll need to look at how we can make the namespace configurable for the initial Kubernetes job. Assuming we can do that, we can pick a Kubernetes namespace at random in StateMachinePlatformExecutor, and submit the bulk import job to that namespace.
#### Testing
We'll need to deploy an instance across multiple AZs with multiple subnets, submit multiple bulk import jobs (e.g. with the system test cluster), and watch them in the console and logs, paying particular attention to the Fargate tasks.
### Dependencies / Blockers
This seems likely to conflict with the following issue, as both work on the same CDK code:
- https://github.com/gchq/sleeper/issues/7068
It's possible that the following issue may also involve similar CDK changes:
- https://github.com/gchq/sleeper/issues/7066
Contributor guide
Research direction
Start with EksBulkImportStack, StateMachinePlatformExecutor, and java/deployment/cdk/src/main/resources/step-functions/run-job.json to trace how Fargate profiles, namespaces, and Spark jobs are configured. Determine how the initial Kubernetes job can receive a configurable namespace, then deploy across multiple subnets and AZs and submit multiple bulk import jobs. Done means jobs are distributed across namespaces and availability zones, verified through the console and logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, kubernetes, spark
- Domain
- backend, cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100