dragonflydb / dragonflydb/dragonfly
Placeholder: Implement QoS and Backpressure for Inbound Slot Migration
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
**Background**
Currently, during cluster re-balancing or slot migration, target nodes might experience significant tail latency spikes ($P99$) for read operations. When a target node is simultaneously serving high-volume GET requests and ingesting new slots, the migration process consumes disproportionate CPU cycles and network bandwidth.
**There is currently a mechanism** to throttle the migration speed on target nodes via `-slot_migration_throttle_us` which I frankly forgot about, so this issue is a placeholder once we actually get actionable feedback from this VERY basic and static mechanism.
The following issue is about potential improvements to make the mechanism faster and better.
** Adaptive Throttling (Backpressure)
1. Today we have much better tooling to recognize long-running fibers.
2. We have background fibers
3. We can potentially come up with additional triggers to recognize whether we have: a) lots of fibers being active - waiting to run, or Dragonfly-specific - the shard_queue has lots of callbacks waiting to run.
The first step would be to observe slots migration happening with `--slot_migration_throttle_us=20` enabled and gather actionable feedback and see what needs to be tuned.
Contributor guide
Assessment
This issue has not been assessed yet.