tikv / tikv/pd

scheduler: limit the total number of scatter-range schedulers to avoid PD CPU overload

Open
#11,124 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Background
PD currently allows an unbounded number of `scatter-range-scheduler` instances to be added via `pd-ctl scheduler add scatter-range-scheduler `. Each active scatter-range scheduler runs its own periodic scheduling loop and region-fit computation. #3594 already showed that `FitRegion` cost is significant per scatter-range scheduler.

### Problem
There is no upper bound on how many scatter-range schedulers can be created. When the total count grows too large (e.g. via automation or misconfiguration repeatedly adding key-range schedulers), the cumulative scheduling/FitRegion cost can drive PD CPU usage very high, degrading other latency-sensitive PD work (region heartbeat processing, leader election, TSO) and overall service quality.

### Proposal
- Add a configurable limit on the total number of concurrently active scatter-range schedulers.
- When `scheduler add scatter-range-scheduler` is invoked and the current count has already reached the limit, reject the creation with a clear error instead of silently accepting it.
- Expose the current count and limit (via `scheduler config` output and/or metrics) so operators can observe usage before hitting the ceiling.

### Alternatives considered
- Leave enforcement to operators manually tracking scheduler counts — no safety net, easy to overshoot.
- Throttle/rate-limit scatter-range scheduler execution instead of capping count — reduces CPU spikes but doesn't bound per-scheduler bookkeeping overhead, and is a larger change than a simple count check.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `pd-ctl scheduler add scatter-range-scheduler ` handling and the code that tracks active scatter-range schedulers. Then inspect the existing `scheduler config` output and metrics paths. Done means additions are rejected at the configured total limit with a clear error, and the current count and limit are observable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.