scheduler: limit the total number of scatter-range schedulers to avoid PD CPU overload
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Background
PD currently allows an unbounded number of `scatter-range-scheduler` instances to be added via `pd-ctl scheduler add scatter-range-scheduler `. Each active scatter-range scheduler runs its own periodic scheduling loop and region-fit computation. #3594 already showed that `FitRegion` cost is significant per scatter-range scheduler.
### Problem
There is no upper bound on how many scatter-range schedulers can be created. When the total count grows too large (e.g. via automation or misconfiguration repeatedly adding key-range schedulers), the cumulative scheduling/FitRegion cost can drive PD CPU usage very high, degrading other latency-sensitive PD work (region heartbeat processing, leader election, TSO) and overall service quality.
### Proposal
- Add a configurable limit on the total number of concurrently active scatter-range schedulers.
- When `scheduler add scatter-range-scheduler` is invoked and the current count has already reached the limit, reject the creation with a clear error instead of silently accepting it.
- Expose the current count and limit (via `scheduler config` output and/or metrics) so operators can observe usage before hitting the ceiling.
### Alternatives considered
- Leave enforcement to operators manually tracking scheduler counts — no safety net, easy to overshoot.
- Throttle/rate-limit scatter-range scheduler execution instead of capping count — reduces CPU spikes but doesn't bound per-scheduler bookkeeping overhead, and is a larger change than a simple count check.
Contributor guide
Research direction
Start by tracing the `pd-ctl scheduler add scatter-range-scheduler ` handling and the code that tracks active scatter-range schedulers. Then inspect the existing `scheduler config` output and metrics paths. Done means additions are rejected at the configured total limit with a clear error, and the current count and limit are observable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100