pingcap / pingcap/tidb

ddl: ADD PARTITION with TiFlash replica causes unnecessary ~2.5s wait per partition

Open
#67,919 0 comments 0 reactions 0 assignees View on GitHub
contribution
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Problem

During `ALTER TABLE ... ADD PARTITION` when the table has a TiFlash replica, the DDL worker enters `StateReplicaOnly` and polls every ~2.5 s waiting for TiFlash replication of the new partition to complete before advancing the partition to public state.

For workloads that write only to TiKV (e.g. TiDB Lightning local-backend bulk ingestion), this wait is completely unnecessary and adds significant latency. At scale — for example 100 range partitions added one at a time — this can add 4+ minutes of pure DDL wait time with zero benefit.

## Root cause

`pkg/ddl/partition.go`, `StateReplicaOnly` case: `checkPartitionReplica` is called unconditionally whenever `TiFlashReplica.Available == true`. If the replica is not yet synced for the new partition the function returns `needRetry=true` and the DDL sleeps for `tiflashCheckTiDBHTTPAPIHalfInterval` (~2.5 s) before retrying.

## Fix

Add a session variable `tidb_skip_tiflash_replica_wait` (default `OFF`, scope `SESSION`) that, when `ON`, causes the DDL worker to skip the `checkPartitionReplica` polling loop entirely and advance the partition to public state immediately. The background TiFlash ticker (`refreshTiFlashTicker`) is still responsible for eventually adding the new partition ID to `TiFlashReplica.AvailablePartitionIDs` once replication completes.

A correctness gap is also fixed: previously, after the `skipWait` check, there was an unconditional block that appended new partition IDs to `TiFlashReplica.AvailablePartitionIDs` when `TiFlashReplica.Available == true`. With `skipWait=true` this block ran and incorrectly marked the partition as TiFlash-available before replication had actually completed. The fix gates this block on `!skipWait`.

## Impact

- Default behavior (`skipWait=false`) is completely unchanged.
- Users running TiDB Lightning or other TiKV-only ingestion pipelines can set `SET SESSION tidb_skip_tiflash_replica_wait = ON` before issuing `ADD PARTITION` DDL to eliminate the wait.
- TiFlash query correctness is preserved: the new partition is not marked available in `AvailablePartitionIDs` until the background ticker confirms replication.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.