apache / apache/accumulo

Investigate how to handle examining files as part of compaction planning

Open
#3,526 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

As part of selecting files to compact for user compactions the compaction selector plugin offered examining [summaries](https://github.com/apache/accumulo/blob/06c55fbea47f7ac376c7c93b48b76529e0d4a474/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java#L60) and [sample data](https://github.com/apache/accumulo/blob/06c55fbea47f7ac376c7c93b48b76529e0d4a474/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/CompactionSelector.java#L71) as part of this decision. This data supported use cases like the [TooManyDeletesSelector](https://github.com/apache/accumulo/blob/06c55fbea47f7ac376c7c93b48b76529e0d4a474/core/src/main/java/org/apache/accumulo/core/client/admin/compaction/TooManyDeletesSelector.java#L66) which could trigger a full tablet compaction when the ratio of deletes in a tablet exceeded a certain threshold.

When the examination ran in tablet servers this examination had much more CPU and memory available than what will be available when running the examination in the manager. In #3513 which moved user compaction from tablet server to the manager this functionality was not implemented because of concerns about resources.

Contributor guide

Open the contributing guide

Research direction

Start with CompactionSelector.java and TooManyDeletesSelector.java, then review the user-compaction changes in #3513. Investigate how summaries and sample data could be examined during manager-side compaction planning under its resource constraints. Done means the project has a decided approach for handling this examination functionality.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.