Add failed compaction counts for specific rfiles.
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Is your feature request related to a problem? Please describe.**
When using external compactions, there is a possibility that a compaction job could contain an rfile that exceeds the resources assigned to a compactor.
If that compactor dies with an OOM issue, there exists no mechanism to indicate that that specific rfile should be moved to a different compaction queue that uses larger compactors.
**Describe the solution you'd like**
When the compaction-coordinator detected a failed compaction job, it should add an errored rfile entry with a count.
At a specific failure threshold, these files should be returned as a "large" compaction criteria.
**Describe alternatives you've considered**
The compaction-coordinator could have knowledge of which queue has "larger" compactor resources and auto submit these compactions.
**Additional context**
conditional mutations are probably needed for this to work
Contributor guide
Research direction
Start by locating the compaction-coordinator entry point and its failed compaction job handling. Read how rfile state is stored and whether conditional mutations are already used; done should include recording per-rfile failure counts and returning files that reach the chosen threshold as "large" compaction criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100