apple / apple/foundationdb

Split small key/value write hotspots

Open
#1,450 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We split shards to resolve write hotspots. The metric that we use to determine when to split is based on bandwidth, ie. the total size of mutations going to a particular storage server.

This means that one can drown a storage server in small key-value pair writes, or with atomic operations, and it's unlikely that data distribution will recognize that a shard is on fire and should be split.

It'd be nice to be able to split based on mutations per second as well as write bandwidth.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the data-distribution logic that measures write bandwidth and decides when a shard should split; the issue does not name files or tests. Determine how mutation rate, including small key-value writes and atomic operations, should be measured alongside bandwidth, then verify that hotspots can trigger splitting based on mutations per second.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.