gchq / gchq/sleeper

Prototype lambda to retry failed bulk import jobs

Open
#7,849 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Description / Background

We have a couple of options we're considering to retry failed bulk import jobs:
1. StepFunctions support for submitting and monitoring
2. A plain lambda on a schedule, checking the state of jobs and retrying when necessary

The first option involves creating a state machine in StepFunctions to submit a bulk import job, that would also detect failures and retry it automatically. This involves significant risk because any time we want to change how we define the state machine, we would have to deploy it with the CDK in order to test it. This would be very time consuming for developers. It would also be difficult to test the retry behaviour at all end to end.

We'd like to prototype the second option to see if it's a viable alternative. We'd like a minimal implementation, but we want to put it live immediately if it turns out to be good enough.

### Technical Notes / Implementation Details

We can create a lambda that runs on a schedule, and follows these steps every time:
1. Scan the bulk import job definitions held in S3
2. Scan the bulk import job statuses held in the tracker
3. For each job that is in S3 but hasn't succeeded yet, check how long it's been since it was written to S3
4. If it's been too long, resubmit it to Spark

This can sit in the existing bulk-import-submitter module, and share code with the bulk import submitter.

We can create a new instance property to enable this, and make the CDK deployment of the lambda conditional on that. We can default it to false for now.

#### Avoid re-detecting the same jobs

When we retry a job, we'll need to be able to tell that we did that and avoid creating duplicate retries.

We have a couple of options for tracking to support this:
1. Metadata on the bulk import job definition that gets re-written to S3
2. Metadata in the job tracker

We can assume we prefer the first option.

We already have some metadata on the job definition in S3, because S3 tracks when a file was written. If we just re-write the same file to S3, that could update that time. This should work as a minimal implementation for this issue.

#### Maximum attempts

It would be good to track how many times we've submitted a job to Spark, and cancel it if it's taken too long. We could use the same sort of metadata described above, to also track the number of attempts.

We don't need to include this in this issue, but it would be a big improvement if it's easy. We could include the number of attempts in the BulkImportJob definition that's written to S3.

#### Job tracker updates

We could consider submitting job tracker updates in this lambda, but we want to leave that out or keep it to a minimum for now.

We'll get a fresh job started update when it's picked up in the new Spark driver. We probably don't need to store an update when we decide to retry a job.

When we decide a job has reached its maximum attempts, it might be easy to update the job tracker to report the failure. If it's not easy, we can leave it out for now. We would need to avoid adding the same update more than once.

#### Failures held in the job tracker

A potential improvement on the initial plan is to detect failures that have been reported to the job tracker, and retry the job quicker instead of waiting for a set period. We want to leave that out for now.

#### Checking state in Spark

A potential improvement is to check the Spark cluster to tell when something has failed there, and retry the job quicker instead of waiting for a set period. We want to leave that out for now.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing bulk-import-submitter module and its shared submitter code, then trace how bulk import definitions are stored in S3, statuses are held in the tracker, and the CDK deployment is configured. Done means a scheduled, conditionally deployed prototype scans stale unsuccessful jobs and resubmits them to Spark without repeatedly retrying the same job; it should be disabled by default.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
backend, cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.