apache / apache/accumulo

Add failed compaction counts for specific rfiles.

Open
#5,024 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
When using external compactions, there is a possibility that a compaction job could contain an rfile that exceeds the resources assigned to a compactor.
If that compactor dies with an OOM issue, there exists no mechanism to indicate that that specific rfile should be moved to a different compaction queue that uses larger compactors.

**Describe the solution you'd like**
When the compaction-coordinator detected a failed compaction job, it should add an errored rfile entry with a count.
At a specific failure threshold, these files should be returned as a "large" compaction criteria.

**Describe alternatives you've considered**
The compaction-coordinator could have knowledge of which queue has "larger" compactor resources and auto submit these compactions.

**Additional context**
conditional mutations are probably needed for this to work

Contributor guide

Open the contributing guide

Research direction

Start by locating the compaction-coordinator entry point and its failed compaction job handling. Read how rfile state is stored and whether conditional mutations are already used; done should include recording per-rfile failure counts and returning files that reach the chosen threshold as "large" compaction criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.