pingcap / pingcap/tidb

lightning: registerTaskToPD causes context deadline exceeded under high parallelism

Open
#67,895 1 comment 0 reactions 0 assignees View on GitHub
contribution
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report / Feature Request

### Problem Summary

When many TiDB Lightning jobs run concurrently against the same TiDB/PD cluster, the `registerTaskToPD` RPC frequently fails with `context deadline exceeded`. PD serialises task registrations via etcd leases, so at high concurrency (e.g. 1,000+ concurrent jobs) the registration queue drains slowly and individual jobs time out before they can register.

### Reproduction

Run 100+ Lightning jobs simultaneously against the same cluster with `pause-pd-scheduler-scope = "table"` (default) or `"global"`. Monitor logs for:

```
context deadline exceeded
```

during the PD task registration phase at import start.

### Expected Behavior

Lightning should provide a way to skip PD scheduler registration entirely when the operator knows the cluster is under controlled high-parallelism ingestion and wants to manage scheduling externally.

### Suggested Solution

Add `pause-pd-scheduler-scope = "off"` as a valid configuration value that skips:
- `registerTaskToPD` (etcd registration under `/tidb/brie/import/lightning/...`)
- `CheckAndPausePdSchedulers` (global scheduler removal)
- `CheckAndFinishRestore` (scheduler restore lifecycle)

The trade-off is that BR log backup cannot detect active Lightning runs. This is acceptable when the operator ensures BR point-in-time restore is not run concurrently with the ETL pipeline.

### Environment

- TiDB version: v8.x+
- Deployment: Many concurrent Lightning instances (e.g. partition-level ETL pipelines)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.