Comcast / Comcast/sirius

Old deletes need to be purged during Compaction

Open
#112 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
296
Forks
48
PR merge metrics
No merged PRs in 30d

Description

Before 'live' compaction, DELETE events older than 7 days (by default) were compacted out during WalTool compaction.

When Sirius switched to using live compaction, that bit of logic was left out. So if a DELETE event is not followed by a subsequent PUT then it lives in the Uberstore forever.

For our usage, this has led to more than 165,000,000 stale DELETE events in the Uberstore. That represents almost 3x the live PUTs and 15gb out of the 40gb Uberstore.

Sirius needs to re-add the ability to purge DELETE events that are older than a specified age while doing compaction.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.