Investigate adjusting sliding window in default compaction planner based on size
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
The default compaction planner operates with the following three constraints when looking for files to compact for system compactions.
1. A max number of files
2. A max files size, this is an optional constraint based on config.
3. compaction ratio.
It attempts to find sets of files that satisfy all three constraints. In its search it has a sliding window based on the [number of files](https://github.com/apache/accumulo/blob/f9897862dd4e6ff4892239ff5ebeb8ed6e34bc68/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java#L474-L481), however the set of files based on size is a [fixed list](https://github.com/apache/accumulo/blob/f9897862dd4e6ff4892239ff5ebeb8ed6e34bc68/core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java#L459-L472). Need to determine if the planner could include total file size in addition to file count in its sliding window.
The first step of work on this is coming up with an example set of files where a sliding window based on the sum of sizes is needed. Need an example that proves this work is needed, something that the current algorithm will not find but an algorithm with a different sliding window technique would find.
**Expected behavior**
Ideally if a set of files exists in a tablet that meet the configured constraints for compaction, then the default compaction planner should be able to fnd that set if its computationally feasible to do so.
Contributor guide
Research direction
Start in core/src/main/java/org/apache/accumulo/core/spi/compaction/DefaultCompactionPlanner.java, especially the fixed size list and file-count sliding window linked in the issue. Construct a file-size example that meets the configured compaction constraints but is missed by the current search. Done means documenting an example that the current algorithm cannot find and a size-aware sliding-window approach could find.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100