apple / apple/foundationdb

Special case shard splitting for sequential insert workloads

Open
#1,322 2 comments 1 reaction 0 assignees View on GitHub
data distribution
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Currently, the logic used to split a shard that has grown too large is to choose a new boundary key that cuts the shard in half. Assuming a vaguely well distributed workload, this is a good choice.

However, workloads that always sequentially append are reasonably common. Most notably: any key that contains a version stamp. In this case, we'd be far better off declaring that the shard should be split by drawing the new boundary as the last currently existing key, and letting the workload proceed into filling up the new, empty shard.

I would imagine that tracking if >99% of write requests are past the ends of one of the byte samples shouldn't be terribly hard to rig up?

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by locating the shard-splitting logic and the byte-sample handling used for write requests. Determine how to detect writes that are over 99% past a sample boundary, then evaluate splitting at the last existing key for sequential workloads. Done means this behavior is implemented and covered by tests without changing ordinary midpoint splitting.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.