apache / apache/accumulo

Consider using DataSketches to precompute quantiles or other values to aid with more rapid split point computation

Open
#3,651 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

DataSketches is useful for precomputing various distribution statistics of data read exactly once. If we use it when we write a file, we could pre-compute things and store it in the file metadata to help make split point computation faster. In order for this to be useful, we would need to make sure we could aggregate the pre-computed statistics across locality groups within a file and across files, so calculation of approximate midpoints can be done very efficiently, only needing to read this pre-computed data to find a suitable midpoint when automatically splitting tablets.

Contributor guide

Open the contributing guide

Research direction

The issue does not name files or tests. Start by locating the file-writing metadata path and the automatic tablet split-point computation, then investigate how statistics could be aggregated across locality groups and files. Done means an agreed design and implementation that stores aggregateable precomputed statistics and uses them for approximate midpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.