apache / apache/accumulo

Allow compactions to delete non shared input files.

Open
#5,387 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**

Many files referenced are only used by a single tablet and these files could be deleted by compaction if this was known. Instead a delete marker is always added for files and GC has to process this delete marker.

**Describe the solution you'd like**

Each files in a tablets metadata could have a shared marker that tracks if more than one tablet references the file.

* When compaction creates a new files it sets shared=false
* When a tablet splits it will set shared=true on any files that go to multiple tablets
* When a table is cloned it will set shared=true in the source table on any files it references in the new table.
* Bulk import could marks files as shared or not depending on if the files go to multiple tablets.
* The fate operation that commits a compaction could either delete the input files or write a delete markers depending on if the files were shared or not.

For this feature to be possible all of the above operations must be able to be done safely using conditional mutations.

The shared marker could be added to the per file metadata that is already stored in the tablet.

**Describe alternatives you've considered**

#2729 may be an alternative if HDFS supports hard links.

**Additional context**

This feature would reduce the work on the Accumulo GC process and avoid storing delete markers. The trade off is that the new shared marker would be required and compaction commit would now be making calls to the namenode to delete files in some cases.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the fate operation that commits a compaction, then inspect tablet file metadata and the split, clone, and bulk-import paths described in the issue. Determine how conditional mutations can safely maintain the shared marker and how compaction distinguishes shared from non-shared inputs. Done means all listed operations preserve correct sharing state and compaction safely deletes eligible files without unnecessary delete markers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.