pingcap / pingcap/tidb

[ddl] Ingest-mode ADD INDEX / ADD PRIMARY KEY roll back on retryable leader-change family instead of retrying

Open
#70,026 1 comment 0 reactions 1 assignee Claimed by @wjhuang2016 View on GitHub
affects-7.5 affects-8.1 affects-8.5 component/ddl found-by-ai severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

> Evidence status: confirmed.

### 1. Minimal reproduce step (Required)

On testbed 8220955 (TiDB v9.0.0-beta.2.pre-1895-g5c9198e948), use a commit-matched failpoint owner fp-tidb built from 5c9198e948. Lower-bridge controls: arm pkg/ingestor/ingestctrl/FailIngestMeta=1*return("notleader") or .../NoLeader=1*return(true) during 100k-row ingest ADD INDEX; logs show retryable Ingest/KV errors and jobs 1548/1551 still synced. Bridge RED: arm pkg/ddl/mockDDLIngestClassifierErr at runIngestReorgJob after runReorgJobAndHandleErr and before isRetryableJobError; 1*return("noleader") on ADD INDEX t4 => job 1557 rollback done; 1*return("notleader") on ADD INDEX t8 => job 1584 rollback done; 1*return("noleader") on ADD PRIMARY KEY tpk1 => job 1563 rollback done; 1*return("regionnotfound") on ADD PRIMARY KEY tpk5 => job 1587 rollback done. Same-environment controls with the failpoint cleared (t5/t9/tpk2/tpk6) all synced.

### 2. What did you expect to see? (Required)

Retryable leader-change-family errors (ErrNoLeader, ErrKVNotLeader, ErrKVRegionNotFound) should remain on a recovery path when they surface during ingest-mode DDL; a transient foreign error should not terminally roll back the job after one hit if the same operation is otherwise valid.

### 3. What did you see instead? (Required)

At the DDL classifier bridge, the user sees region has no leader / mock ddl ingest classifier not leader / mock ddl ingest classifier region not found, and jobs 1557, 1563, 1584, and 1587 all end rollback done. Owner logs show Unknown error class [class=KV/Ingest] and run reorg job failed, convert job to rollback. The same family below the bridge inside ingestctrl stayed GREEN.

### 4. What is your TiDB version? (Required)

TiDB v9.0.0-beta.2.pre-1895-g5c9198e948 ingest online DDL path

Likely root cause and fix direction

The inner ingestctrl worker treats ErrKVNotLeader, ErrNoLeader, and ErrKVRegionNotFound as retryable via lightning/common.IsRetryableError, but outer DDL pkg/ddl/index.go:isRetryableError/isRetryableJobError only recognizes DDL-native retryable codes/messages. Once a foreign Ingest/KV error crosses from runReorgJobAndHandleErr into runIngestReorgJob, it falls out of the retryable set after unknown-class normalization and is fatalized into rollback.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.