gchq / gchq/sleeper

Compaction reliability

Open
#7,314 0 comments 0 reactions 0 assignees View on GitHub
epic
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want compaction jobs to run deterministically and reliably, so that I know my data will be processed and won't be lost.

### Description / Background

We'd like to ensure that:
- Data scheduled into a compaction job will always be written to a new file.
- The new file replaces the old file atomically, such that no data is lost or duplicated.
- If data cannot be processed, it should be transparent to the user why this was the case, and what they can do about it.
- If a compaction job cannot be processed for some temporary reason, the underlying data will still be correctly processed.

### Technical Notes / Implementation Details

See sub-issues. We may make notes here on potential issues that have not yet been raised.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.