Consider using DataSketches to precompute quantiles or other values to aid with more rapid split point computation
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 487
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 13
Description
DataSketches is useful for precomputing various distribution statistics of data read exactly once. If we use it when we write a file, we could pre-compute things and store it in the file metadata to help make split point computation faster. In order for this to be useful, we would need to make sure we could aggregate the pre-computed statistics across locality groups within a file and across files, so calculation of approximate midpoints can be done very efficiently, only needing to read this pre-computed data to find a suitable midpoint when automatically splitting tablets.
Contributor guide
Research direction
The issue does not name files or tests. Start by locating the file-writing metadata path and the automatic tablet split-point computation, then investigate how statistics could be aggregated across locality groups and files. Done means an agreed design and implementation that stores aggregateable precomputed statistics and uses them for approximate midpoints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100