pingcap / pingcap/tidb

ddl: add feedback-driven adaptive backfill throttling

Open
#70,410 1 comment 0 reactions 1 assignee Claimed by @expxiaoli View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

### Problem

Large DDL backfill jobs currently depend on relatively static values for reorg worker count, batch size, and maximum write speed. Operators must manually balance DDL throughput against online workload latency and cluster resource usage.

Static settings cannot adapt when workload or cluster pressure changes:

- Conservative settings protect online traffic but make DDL unnecessarily slow.
- Aggressive settings can increase TiDB memory and GC pressure.
- Heavy backfill writes can increase TiKV disk latency, compaction backlog, flow control, and scheduler latency.
- Multiple concurrent DDL jobs can independently use their configured limits and collectively overload shared TiKV resources.
- Different backfill phases may not support consistent runtime adjustment.

TiDB already supports changing some reorg metadata and distributed-task resources while a job is running. However, changes are externally driven, stage coverage is incomplete, and concurrent jobs are not coordinated by a cluster-wide feedback controller.

### Proposed feature

Add an opt-in controller that dynamically adjusts the effective DDL backfill worker count, batch size, and maximum write speed according to current TiDB, TiKV, and online-workload pressure.

#### Cluster-wide budget

- Maintain an aggregate DDL backfill budget for shared cluster resources.
- Allocate the budget fairly across concurrent jobs and avoid starvation.
- Recalculate allocations when jobs start, finish, pause, resume, or change phase.
- Treat existing reorg settings as administrator-defined upper bounds.

#### Feedback signals

Use multiple signals rather than CPU alone:

- TiDB memory headroom, GC pressure, CPU, backfill retries, and OOM risk;
- TiKV disk latency, pending compaction bytes, flow control, write stalls, scheduler latency, and busy responses;
- DDL throughput, phase, and backoff rate;
- an optional online query-latency or external SLO pressure signal.

The first version should at least cover TiDB memory pressure and TiKV I/O/compaction pressure.

#### Control strategy

- Back off quickly when online latency exceeds its guardrail or the cluster enters a high-pressure state.
- Increase load slowly after headroom remains stable for an observation window.
- Use hysteresis, cooldown periods, bounded steps, and smoothed measurements to avoid oscillation.
- Reduce maximum write speed first for TiKV write or compaction pressure.
- Reduce worker count and batch size for TiDB memory or excessive concurrency pressure.
- Increase only one control dimension at a time unless stable headroom is clearly available.

#### Backfill coverage

Support live adjustment, or explicitly report that it is unsupported, for:

- transactional reorg and reorg-data operations such as `MODIFY COLUMN`;
- local-ingest index backfill;
- distributed/global-sort backfill;
- temporary-index merge.

The implementation should reuse existing `DDLReorgMeta` and distributed-task modification mechanisms where possible and add missing runtime-update hooks for uncovered phases.

### Safety and compatibility

- Keep adaptive control disabled by default initially.
- Preserve existing behavior when it is disabled.
- Fall back to the last safe allocation or static configuration when feedback is unavailable or stale.
- Preserve safe controller behavior across owner changes, retries, pause/resume, and failover.
- Allow explicit administrator overrides to take precedence.
- Ensure adjustments affect performance only and do not change DDL correctness, checkpoint consistency, or transactional semantics.

### Observability

Expose through metrics, logs, and DDL job inspection:

- configured and effective worker count, batch size, and write speed;
- aggregate budget and each job's allocation;
- controller state: increase, decrease, hold, override, or fallback;
- the signal and reason for the latest adjustment;
- whether the current phase supports live adjustment.

### Acceptance criteria

1. A large index backfill can use idle headroom while staying within configured TiDB memory and online-latency guardrails.
2. Concurrent DDL jobs share one aggregate budget and cannot each consume the full cluster limit.
3. Backfill load is automatically reduced before sustained TiKV compaction, flow-control, or scheduler pressure causes a severe latency regression.
4. Reorg-data operations continue making progress while the controller protects the online workload.
5. Effective parameters change without restarting the DDL job and remain safe after phase changes or failover.
6. Metrics and job status explain every adjustment.
7. Tests cover bounds, hysteresis, stale feedback, overrides, concurrent jobs, phase transitions, and recovery.
8. No DDL correctness, cancellation, checkpoint, or recovery regression is introduced.

### Non-goals

This feature does not:

- fix application deadlocks, lock ordering, same-key hotspots, or oversized transactions;
- eliminate the required scan and write work of a DDL operation;
- replace disk-capacity planning;
- hide memory leaks or unbounded buffering defects;
- guarantee an online latency SLO when non-DDL traffic already saturates the cluster.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.