dragonflydb / dragonflydb/dragonfly

Placeholder: Implement QoS and Backpressure for Inbound Slot Migration

Open
#7,248 1 comment 0 reactions 1 assignee Claimed by @BorysTheDev View on GitHub
enhancement minor
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

**Background**
Currently, during cluster re-balancing or slot migration, target nodes might experience significant tail latency spikes ($P99$) for read operations. When a target node is simultaneously serving high-volume GET requests and ingesting new slots, the migration process consumes disproportionate CPU cycles and network bandwidth.

**There is currently a mechanism** to throttle the migration speed on target nodes via `-slot_migration_throttle_us` which I frankly forgot about, so this issue is a placeholder once we actually get actionable feedback from this VERY basic and static mechanism.

The following issue is about potential improvements to make the mechanism faster and better.

** Adaptive Throttling (Backpressure)

1. Today we have much better tooling to recognize long-running fibers.
2. We have background fibers
3. We can potentially come up with additional triggers to recognize whether we have: a) lots of fibers being active - waiting to run, or Dragonfly-specific - the shard_queue has lots of callbacks waiting to run.

The first step would be to observe slots migration happening with `--slot_migration_throttle_us=20` enabled and gather actionable feedback and see what needs to be tuned.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.