pingcap / pingcap/tidb

ddl: upgrade from v7.5.7 to master during `ADD INDEX` cause data inconsistency

Open
#70,779 1 comment 0 reactions 0 assignees View on GitHub
affects-8.5 component/ddl severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

Run `ADD INDEX` with local sort on TiDB v7.5.7 and perform a rolling upgrade to master during reorg. After the job was resumed after upgrading, `ADD INDEX` finished with inconsistent data.

### 2. What did you expect to see? (Required)

`ADD INDEX` should finish without error and data inconsistency.

### 3. What did you see instead (Required)

```text
ERROR 8223 (HY000): data inconsistency in table: nto_ssmx_bill_detail,
index: idx1, handle: 360287970197478193,
index-values:"" != record-values:"handle: 360287970197478193, ..."
```

### 4. What is your TiDB version? (Required)

v7.5.7 -> master

### Analysis

In v7.5.7, the checkpoint was stored with `PhysicalID`, `StartKey`, and `EndKey`, which represent the successful imported position, and the resumed node should restart from this this position.

But the refactor in #54747 removed `StartKey` and `EndKey` and changed the recovery logic. Recovery only reads `GlobalSyncKey` and `PhysicalID`. When the pid from the checkpoint differs from `reorgInfo.PhysicalTableID`, the checkpoint was discarded.

So when new node restart the job, the old checkpoint `physical_id=114` was ignored, and ranges between 114 and `reorgInfo.PhysicalTableID` are missed.

TiDB log

Old node

```text
[2026/09/01 07:00:21.181 +08:00] [INFO] [checkpoint.go:407] ["update checkpoint"] [category=ddl-ingest] [jobID=250] [indexIDs="[12]"] ["local checkpoint"=] ["global checkpoint"=] ["global physical ID"=114] []
...
[2026/09/01 07:01:47.766 +08:00] [INFO] [job_table.go:414] ["handle ddl job failed"] [category=ddl] [error="[ddl:8201]TiDB server is not a DDL owner"] [job="ID:250, Type:add index, State:running, SchemaState:write reorganization, SchemaID:181, TableID:113, RowCount:17168264, ArgLen:6, start time: 2026-09-01 06:56:24.44 +0800 CST, Err:[ddl:-1]peer 1967, store 5, region 1964, epoch conf_ver:5 version:99 , when send data: rpc error: code = Canceled desc = context canceled, ErrCount:2, SnapshotVersion:468770353247682562, UniqueWarnings:0"]
```

New node

```text
[2026/09/01 07:01:56.513 +08:00] [INFO] [job_scheduler.go:749] ["resume physical table ID from checkpoint"] [category=ddl-ingest] [jobID=250] ["global sync key"=] ["checkpoint physical ID"=114]
[2026/09/01 07:01:56.513 +08:00] [INFO] [index.go:2944] ["start to add table index"] [category=ddl] [job="ID:250, Type:add index, State:running, SchemaState:write reorganization, SchemaID:181, TableID:113, RowCount:16947517, ArgLen:6, start time: 2026-09-01 06:56:24.44 +0800 CST, Err:, ErrCount:1, SnapshotVersion:468770353247682562, Version: v1, UniqueWarnings:0"] [reorgInfo="CurrElementType:_idx_,CurrElementID:12,StartKey:7480000000000000af5f728c00000000989681,EndKey:7480000000000000af5f72bc0000000098962200,First:false,PhysicalTableID:175,Ingest mode:true"]
[2026/09/01 07:01:56.514 +08:00] [INFO] [env.go:123] ["the ingest sorted directory"] [category=ddl-ingest] ["data path"=/tmp/tidb/tmp_ddl-4000]
```

Contributor guide

Open the contributing guide

Research direction

Start with checkpoint.go and job_scheduler.go, tracing how the physical ID and checkpoint are restored when an ADD INDEX job resumes; then read the ADD INDEX flow in index.go. Reproduce the v7.5.7-to-master rolling upgrade during reorganization and verify that the resumed job processes every range without reporting inconsistent index data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.