apple / apple/foundationdb

Data rebalancing may get stuck when Redwood is too fast in write

Open
#3,664 1 comment 0 reactions 1 assignee Claimed by @etschannen View on GitHub
data distribution
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

In a fast restore experiment, we (Evan and I) noticed that a two-member SS team gets 200MB/s (?) input (write) bytes and ~30MB/s output (read) bytes. The cluster uses Redwood storage engine with double replication.

Data distribution cannot finish relocating shards for rebalancing the load, because destination SSs cannot catch up in reading/moving data away from the hammered hot SSs.

We may need to experiment with the write-heavy workload for Redwood and confirm if this can happen. If so, we need to see if the tag-throttling feature can solve this situation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.