dragonflydb / dragonflydb/dragonfly
Throttle full sync replication on master
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
When testing replication with SSD tiered replica, we saw that the latency on the master jumps.
The reason for this is that the back pressure from the replica back propagates to the master, and then write commands can not proceed with writes as they wait on socket even with very modest write traffic, as the majority of the bandwidth is used by the snapshotting process.
Recently we had a similar problem with outgoing slots migrations and I fixed it inside `RestoreStreamer::Run` by adding the `(pending_buf_.Size() >= replication_stream_output_limit_cached / 3)` check that postpones the migrations if the socket is overfilled. It was successful to make slot migrations less intrusive to the outgoing writes (SETs).
It's not straightforward to do such check with full sync replication as we do not employ the async interface there. But I would love adding some heuristic that regulates snapshotting pace so that outgoing writes won't be throttled. Wth async interface we have an explicit `JournalStreamer::ThrottleIfNeeded` but with sync interface it happens in the OS level when the networking output buffer is full and it pushed the packets out.
## Read latency
Another problem that also requires throttling of the writes inside full sync - outgoing bandwidth limits of the host. When replication happens it easily reaches GB/s bandwidth that can saturate the network limits of the master host. When it happens then with read heavy workloads, the outbound bandwidth is throttled by the cloud and the latency of read requests (MGETs) skyrockets as well. I would like us to introduce the `outgoing_replication_bandwidth_limit` flag that would allow us to limit the pace of the snapshotting (but not the outgoing writes).
Contributor guide
Research direction
Start by comparing the full-sync replication path with RestoreStreamer::Run and JournalStreamer::ThrottleIfNeeded, which the issue identifies as related throttling points. Trace how the synchronous snapshot reaches the networking output buffer and determine where an outgoing_replication_bandwidth_limit flag would apply. Done means snapshotting is paced without throttling outgoing writes, while read latency and replication behavior are covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100