gchq / gchq/sleeper

Bulk import reliability

Open
#7,313 0 comments 0 reactions 0 assignees View on GitHub
epic
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want ingest and bulk import jobs to run deterministically and reliably, so that I know my data will be processed and won't be lost.

### Description / Background

We'd like to ensure that data submitted to the system as an ingest or bulk import job will always be either added to the Sleeper table, or be determined to be invalid and refused.

When a job is refused, it should be transparent to the user why this was the case, and it should be a reason the user would agree with.

### Technical Notes / Implementation Details

See sub-issues. We may make notes here on potential issues that have not yet been raised.

Here's a potential issue we could pick up next:
- https://github.com/gchq/sleeper/issues/5912

There's more information on next steps on the following parent ticket:
- https://github.com/gchq/sleeper/issues/7065

These seem like the most important to focus on in general:
- https://github.com/gchq/sleeper/issues/5731
- https://github.com/gchq/sleeper/issues/6416

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing sub-issues 5912, 5731, and 6416, then read parent issue 7065; the work is complete only when ingest and bulk import jobs are deterministic, reliably added or refused, and refusals provide an acceptable reason.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.