apache / apache/accumulo

Improve scalability of compaction coordinator

Open
#6,279 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**

After the changes in #6217 compactors will likely do the following actions to run a compaction

1. Find the coordinator process for their compactor resource group
2. Make an RPC to request a job from that coordinator. On the server side this will:
1. Pull the next job from an in memory queue
2. Perform metadata table operations to reserve the files for compaction
3. Run the compactoin
4. Make an RPC to the correct coordinator to complete the compaction. This will initiate a fate operation which involves a write to the fate table.

**Describe the solution you'd like**

The operation above to write to the metadata table and the fate table all happens on one coordinator for a given compaction resource group. So if a single compaction resource group has a lot of compactor processes then they will funnel all writes through that single process. One way to spread this out is to allow on any coordinator to process these writes. This would primarily be accomplished by breaking the RPC to get a job into two RPCs, one to get a job and another to reserve a job.

1. Find the coordinator process for their resource group
2. Make an RPC to request a job from that coordinator. This is quick RPC that only pulls a job out of an in memory priority queue. The job returned will not be reserved.
3. Make an RPC request to any coordinator to reserve the job. This will do the conditional metadata table writes on the server side and fail if the job can no longer be reserved.
4. Run the compaction job if it was reserved, otherwise look for a new job.
5. Make an RPC request to any coordinator to complete the job which will write to the fate table on the server side.

**Additional context**

The goal of this change would be to make compactions scale well as the number of compactors increases by taking the following actions as needed.

* Increasing the number of manager processes
* Increasing the number of metadata tservers and splitting the metadata table
* Increasing the number of fate tservers and splitting the fate table.

By adding an RPC this would increase the latency of a single compaction a bit, but hopefully it would allow really high throughput for the entire system.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the changes in #6217 and tracing the current compaction coordinator flow for job requests, metadata-table reservation, and fate-table completion. Map the coordinator RPCs and their interactions with manager processes, metadata tservers, and fate tservers; the work is done when these operations can be distributed across coordinators while preserving compaction correctness and improving throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.