apache / apache/accumulo-fluo

Consider placing notifications in separate table

Open
#1,023 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
200
Forks
73
PR merge metrics
No merged PRs in 30d

Description

Fluo's notifications are currently placed in a separate locality group. This means when scanning for notifications, user data is not read. Notifications can only be garbage collected with a full major compaction. However full major compactions do not happen often. This means that over time notification delete markers build up. This build up over time makes scanning for notifications more expensive.

Work was done for #457 to drop delete marker for notifications that were inserted and processed before an Accumulo minor compaction happens. The work in #457 also has a chance of dropping delete markers on partial compactions. However if a delete marker makes it to the largest files, it may be there for a long time. The effectiveness of #457 is inversely proportional to the average time it takes to process a notification. As the average time goes up, the effectiveness goes down. If too many delete markers have built up, then the best work around is a full compaction of the table.

If notifications were in a separate table, then the full compaction could be cheap. Also the work done for [ACCUMULO-4500](https://issues.apache.org/jira/browse/ACCUMULO-4500) could be leveraged to automatically make smart decisions about when to compact based on the number of notifications vs the number of delete markers.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by reviewing the work in #457 and the ACCUMULO-4500 reference, then trace how notifications are stored and compacted. Done should mean notifications can be compacted cheaply in a separate table and delete-marker buildup is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.