apache / apache/accumulo

RFile writes should utilize multiple threads

Open
#4,124 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
The write operations to an rfile are serialized. When writing large rfiles in map reduce jobs this can produces very large tales to the jobs. The bottleneck is often compression rather than i/o.

**Describe the solution you'd like**
Utilizing multiple threads to process multiple blocks in parallel could dramatically improve write performance. Having a dedicated thread to write completed blocks in order would still be necessary, but should be possible. This could be scaled based on available memory for buffering.

**Describe alternatives you've considered**
Adding pipelines to the existing code could be a smaller lift, and have a big performance improvement as well.

Contributor guide

Open the contributing guide

Research direction

Start by locating the RFile write path and the code that compresses and writes blocks, then trace how map-reduce jobs serialize those operations. Compare the existing serialized path with the proposed parallel block processing and ordered output design. Done should include a measured write-performance improvement without changing block order or correctness.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.