Special case shard splitting for sequential insert workloads
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Currently, the logic used to split a shard that has grown too large is to choose a new boundary key that cuts the shard in half. Assuming a vaguely well distributed workload, this is a good choice.
However, workloads that always sequentially append are reasonably common. Most notably: any key that contains a version stamp. In this case, we'd be far better off declaring that the shard should be split by drawing the new boundary as the last currently existing key, and letting the workload proceed into filling up the new, empty shard.
I would imagine that tracking if >99% of write requests are past the ends of one of the byte samples shouldn't be terribly hard to rig up?
Contributor guide
Research direction
The issue names no files or tests; start by locating the shard-splitting logic and the byte-sample handling used for write requests. Determine how to detect writes that are over 99% past a sample boundary, then evaluate splitting at the last existing key for sequential workloads. Done means this behavior is implemented and covered by tests without changing ordinary midpoint splitting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100