gchq / gchq/sleeper

Split EKS Spark jobs across subnets/AZs

Open
#7,070 0 comments 0 reactions 0 assignees View on GitHub
enhancement needs-refinement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper running bulk import on EKS, I want different bulk import jobs to be spread across availability zones, so that I can take advantage of capacity in all availability zones, and if one availability zone is unavailable jobs will still be processed.

### Description / Background

Under parent issue:
- https://github.com/gchq/sleeper/issues/7065

Currently in EksBulkImportStack we choose one subnet to run bulk import jobs in, and they all run there. A subnet is in a single availability zone, so that means all bulk import jobs will run in one availability zone.

We'd like to run jobs across all subnets.

### Technical Notes / Implementation Details

Currently EksBulkImportStack creates one Fargate profile, associated with one Kubernetes namespace.

We could create a separate Fargate profile and Kubernetes namespace for each subnet, and submit bulk import jobs to all of them.

There are two places where we configure which Kubernetes namespace is used. For the pods created by Spark, we set `spark.kubernetes.namespace` in StateMachinePlatformExecutor. For the initial Kubernetes job to submit to Spark, the namespace is set in the StepFunctions definition in `java/deployment/cdk/src/main/resources/step-functions/run-job.json`. That gets used in EksBulkImportStack.

We'll need to look at how we can make the namespace configurable for the initial Kubernetes job. Assuming we can do that, we can pick a Kubernetes namespace at random in StateMachinePlatformExecutor, and submit the bulk import job to that namespace.

#### Testing

We'll need to deploy an instance across multiple AZs with multiple subnets, submit multiple bulk import jobs (e.g. with the system test cluster), and watch them in the console and logs, paying particular attention to the Fargate tasks.

### Dependencies / Blockers

This seems likely to conflict with the following issue, as both work on the same CDK code:
- https://github.com/gchq/sleeper/issues/7068

It's possible that the following issue may also involve similar CDK changes:
- https://github.com/gchq/sleeper/issues/7066

Contributor guide

Open the contributing guide

Research direction

Start with EksBulkImportStack, StateMachinePlatformExecutor, and java/deployment/cdk/src/main/resources/step-functions/run-job.json to trace how Fargate profiles, namespaces, and Spark jobs are configured. Determine how the initial Kubernetes job can receive a configurable namespace, then deploy across multiple subnets and AZs and submit multiple bulk import jobs. Done means jobs are distributed across namespaces and availability zones, verified through the console and logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, kubernetes, spark
Domain
backend, cloud, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.