Bulk import reliability
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a user of Sleeper, I want ingest and bulk import jobs to run deterministically and reliably, so that I know my data will be processed and won't be lost.
### Description / Background
We'd like to ensure that data submitted to the system as an ingest or bulk import job will always be either added to the Sleeper table, or be determined to be invalid and refused.
When a job is refused, it should be transparent to the user why this was the case, and it should be a reason the user would agree with.
### Technical Notes / Implementation Details
See sub-issues. We may make notes here on potential issues that have not yet been raised.
Here's a potential issue we could pick up next:
- https://github.com/gchq/sleeper/issues/5912
There's more information on next steps on the following parent ticket:
- https://github.com/gchq/sleeper/issues/7065
These seem like the most important to focus on in general:
- https://github.com/gchq/sleeper/issues/5731
- https://github.com/gchq/sleeper/issues/6416
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing sub-issues 5912, 5731, and 6416, then read parent issue 7065; the work is complete only when ingest and bulk import jobs are deterministic, reliably added or refused, and refusals provide an acceptable reason.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100