Split small key/value write hotspots
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
We split shards to resolve write hotspots. The metric that we use to determine when to split is based on bandwidth, ie. the total size of mutations going to a particular storage server.
This means that one can drown a storage server in small key-value pair writes, or with atomic operations, and it's unlikely that data distribution will recognize that a shard is on fire and should be split.
It'd be nice to be able to split based on mutations per second as well as write bandwidth.
Contributor guide
Research direction
Start by tracing the data-distribution logic that measures write bandwidth and decides when a shard should split; the issue does not name files or tests. Determine how mutation rate, including small key-value writes and atomic operations, should be measured alongside bandwidth, then verify that hotspots can trigger splitting based on mutations per second.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100