ddl: upgrade from v7.5.7 to master during `ADD INDEX` cause data inconsistency
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
Run `ADD INDEX` with local sort on TiDB v7.5.7 and perform a rolling upgrade to master during reorg. After the job was resumed after upgrading, `ADD INDEX` finished with inconsistent data.
### 2. What did you expect to see? (Required)
`ADD INDEX` should finish without error and data inconsistency.
### 3. What did you see instead (Required)
```text
ERROR 8223 (HY000): data inconsistency in table: nto_ssmx_bill_detail,
index: idx1, handle: 360287970197478193,
index-values:"" != record-values:"handle: 360287970197478193, ..."
```
### 4. What is your TiDB version? (Required)
v7.5.7 -> master
### Analysis
In v7.5.7, the checkpoint was stored with `PhysicalID`, `StartKey`, and `EndKey`, which represent the successful imported position, and the resumed node should restart from this this position.
But the refactor in #54747 removed `StartKey` and `EndKey` and changed the recovery logic. Recovery only reads `GlobalSyncKey` and `PhysicalID`. When the pid from the checkpoint differs from `reorgInfo.PhysicalTableID`, the checkpoint was discarded.
So when new node restart the job, the old checkpoint `physical_id=114` was ignored, and ranges between 114 and `reorgInfo.PhysicalTableID` are missed.
TiDB log
Old node
```text
[2026/09/01 07:00:21.181 +08:00] [INFO] [checkpoint.go:407] ["update checkpoint"] [category=ddl-ingest] [jobID=250] [indexIDs="[12]"] ["local checkpoint"=] ["global checkpoint"=] ["global physical ID"=114] []
...
[2026/09/01 07:01:47.766 +08:00] [INFO] [job_table.go:414] ["handle ddl job failed"] [category=ddl] [error="[ddl:8201]TiDB server is not a DDL owner"] [job="ID:250, Type:add index, State:running, SchemaState:write reorganization, SchemaID:181, TableID:113, RowCount:17168264, ArgLen:6, start time: 2026-09-01 06:56:24.44 +0800 CST, Err:[ddl:-1]peer 1967, store 5, region 1964, epoch conf_ver:5 version:99 , when send data: rpc error: code = Canceled desc = context canceled, ErrCount:2, SnapshotVersion:468770353247682562, UniqueWarnings:0"]
```
New node
```text
[2026/09/01 07:01:56.513 +08:00] [INFO] [job_scheduler.go:749] ["resume physical table ID from checkpoint"] [category=ddl-ingest] [jobID=250] ["global sync key"=] ["checkpoint physical ID"=114]
[2026/09/01 07:01:56.513 +08:00] [INFO] [index.go:2944] ["start to add table index"] [category=ddl] [job="ID:250, Type:add index, State:running, SchemaState:write reorganization, SchemaID:181, TableID:113, RowCount:16947517, ArgLen:6, start time: 2026-09-01 06:56:24.44 +0800 CST, Err:, ErrCount:1, SnapshotVersion:468770353247682562, Version: v1, UniqueWarnings:0"] [reorgInfo="CurrElementType:_idx_,CurrElementID:12,StartKey:7480000000000000af5f728c00000000989681,EndKey:7480000000000000af5f72bc0000000098962200,First:false,PhysicalTableID:175,Ingest mode:true"]
[2026/09/01 07:01:56.514 +08:00] [INFO] [env.go:123] ["the ingest sorted directory"] [category=ddl-ingest] ["data path"=/tmp/tidb/tmp_ddl-4000]
```
Contributor guide
Research direction
Start with checkpoint.go and job_scheduler.go, tracing how the physical ID and checkpoint are restored when an ADD INDEX job resumes; then read the ADD INDEX flow in index.go. Reproduce the v7.5.7-to-master rolling upgrade during reorganization and verify that the resumed job processes every range without reporting inconsistent index data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100