evict-slow-store: allow draining multiple nodes from the same availability zone
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 33
Description
## Enhancement Task
### Background
The evict-slow-store scheduler transfers leaders away from stores whose disk slow-score exceeds a threshold. It has a deliberate safety guard: if more than one store is slow at the same time, it does nothing.
This is by design — multiple stores degrading simultaneously is a signal of broader cluster instability. Draining leaders in an already-unstable cluster risks making things worse, so the conservative choice is to hold.
### Problem
In cloud environments, an Availability Zone is a single failure domain. A bad deployment, a datacenter power event, or a batch of faulty hardware can cause every node in one AZ to degrade together — while nodes in other AZs remain perfectly healthy. In this situation, all the slow stores share the same AZ label and the cross-zone topology is intact.
The current guard cannot distinguish this case from genuine cluster-wide instability. It sees multiple slow stores and stops, leaving the degraded AZ serving leaders it cannot handle efficiently, when the right action is to drain the whole AZ and let the healthy zones take over.
### Proposal
When all slow stores belong to the same isolation zone, treat the event as an AZ-level failure and drain them all — subject to guards that preserve the original protection against cluster-wide instability:
1. Uniform placement-rule isolation.
Read IsolationLevel from every active placement rule via RuleManager.GetAllRules(). Group eviction only activates when every rule enforces the same non-empty isolation level. If any rule differs or is empty, fall back to the original do-nothing behavior. This ensures the zone concept is actually encoded in the cluster's placement rules, not assumed.
2. All slow stores share one AZ.
Every slow store must carry the same value for the isolation label. If slow stores span two zones, do nothing — this is the genuine multi-domain instability the original guard was designed for.
3. Enough healthy zones remain.
Only drain an AZ if at least 2 other distinct zones still have a healthy (serving, non-slow, leader-transfer-allowed) store. This prevents draining when the topology is too degraded to absorb the load shift.
4. Freeze if a second zone goes slow.
While draining a zone, if any slow score appears in a store outside that zone, stop adding new stores to the eviction set. The situation has changed from "one bad AZ" to "multiple failing domains" — revert to conservative behavior.
5. Group recovery.
Release the drained AZ only once every evicted store has recovered for the configured recovery-duration. The group is treated as a unit: one flapping node does not block recovery of the others, but the whole group must be healthy before leaders flow back.
Relationship to existing slow-store mechanisms
PD has two other mechanisms that partially overlap with this change, and both are intentionally left untouched.
evict-slow-trend detects performance degradation by trend (latency increase, QPS drop) rather than absolute score, and already has a SlowStoreEvictingAffectedStoreRatioThreshold guard (default 0.3) that caps how much of the cluster can be evicted at once. The multi-store guard in that scheduler works differently and is not part of this change.
Network slow store (the secondary path inside evict-slow-store) detects pairwise network degradation between TiKV nodes. It currently caps eviction at one store at a time (defaultMaxNetworkSlowStore = 1) through a separate mechanism: a two-phase pause→evict flow with per-store timers. A zone-level network incident (e.g. a bad switch) could in principle benefit from the same AZ-aware grouping, but the detection logic is more complex and the risk of false positives is higher — that extension is left for a follow-up.
In short
The original guard was designed to protect against an unstable cluster. A full-AZ degradation in a cloud environment is a different event — a known failure domain, not a symptom of cluster-wide problems. This change lets PD tell the difference.
Contributor guide
Research direction
Start by locating the evict-slow-store scheduler and its existing multi-store guard, then inspect RuleManager.GetAllRules() and the current leader-draining recovery timers. Trace how slow scores, isolation labels, healthy stores, and leader-transfer permissions are evaluated. Done means same-zone stores can drain as a group under all listed safety guards, while multi-zone instability preserves the original no-op behavior and recovery is coordinated for the group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100