Create jobs with arbitrary file count atomically in ingest batcher
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Background
We found this bug with the batcher when you try to create an ingest job with more than 50 files:
- https://github.com/gchq/sleeper/issues/1120
This was due to a limit of 100 items in a DynamoDB transaction. We fixed that by breaking the atomicity of the job assignment in the ingest batcher store.
### Description
We'd like the ingest batcher to be able to create ingest jobs atomically, even with more files than fit in a DynamoDB transaction.
### Analysis
One way to apply the update would be to save a record which contains a list of all the files that were assigned to the job, as an event that that job was created. That could be saved atomically, but we'd still have the state of the file records that's not updated.
We could combine that record with the separate records for the state of the files, by ensuring that we query the transaction log every time we load the state of the files. Once the job creation event has been saved, any readers should see that as they read it alongside the rest of the state. At that point we can actually update the state of the files to apply that event, and then we can delete the event from the transaction log.
If the update to the state of the files fails to apply, we can still just move on as that update will still be seen as the event has been applied in the transaction log. We can ensure those state updates will be applied next time we try to apply the transaction log, by leaving the event there.
Contributor guide
Assessment
This issue has not been assessed yet.