gchq / gchq/sleeper

Create jobs with arbitrary file count atomically in ingest batcher

Open
#1,140 0 comments 0 reactions 0 assignees View on GitHub
enhancement ingest-module
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Background

We found this bug with the batcher when you try to create an ingest job with more than 50 files:

- https://github.com/gchq/sleeper/issues/1120

This was due to a limit of 100 items in a DynamoDB transaction. We fixed that by breaking the atomicity of the job assignment in the ingest batcher store.

### Description

We'd like the ingest batcher to be able to create ingest jobs atomically, even with more files than fit in a DynamoDB transaction.

### Analysis

One way to apply the update would be to save a record which contains a list of all the files that were assigned to the job, as an event that that job was created. That could be saved atomically, but we'd still have the state of the file records that's not updated.

We could combine that record with the separate records for the state of the files, by ensuring that we query the transaction log every time we load the state of the files. Once the job creation event has been saved, any readers should see that as they read it alongside the rest of the state. At that point we can actually update the state of the files to apply that event, and then we can delete the event from the transaction log.

If the update to the state of the files fails to apply, we can still just move on as that update will still be seen as the event has been applied in the transaction log. We can ensure those state updates will be applied next time we try to apply the transaction log, by leaving the event there.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.