gchq / gchq/sleeper

Apply filters when files are not otherwise compacted

Open
#6,606 0 comments 0 reactions 0 assignees View on GitHub
enhancement under-discussion
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want an age off filter to apply even when new data is not added to the table, so that the data is deleted.

### Description / Background

If I configure an age-off filter, I expect it to eventually delete all data that ages off. Currently if no compactions happen the underlying data is never deleted.

When new data is added to a Sleeper table, this can trigger compaction, but then the table will reach a stable state where no further compactions occur. If no new data is added, that usually happens much faster than data would age off.

We'd like to automatically apply an age-off filter to existing data even when no new data is added.

### Acceptance Criteria

**Given** I configure an age off filter on a Sleeper table
**When** I add data to the table
**And** the table settles into a state not needing compaction before any data ages off
**Then** the data will still be deleted after it ages off

### Technical Notes / Implementation Details

There are some options for this:
- Schedule a regular full compaction on a schedule
- When creating compaction jobs, look for old file references and ensure they are compacted

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is named. Start by tracing how compaction jobs are created and how age-off filters are applied to existing file references, then compare the scheduling and compaction-job options described here. Done means aged-off data is deleted even when the table receives no new data after settling.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.