add index during upgrade from v8.4.0 to v8.5.6-pre may leave index/data inconsistency after jobs are synced
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce path from TCMS
This issue comes from TCMS plan execution `8107798`:
- Plan execution: https://tcms.pingcap.net/dashboard/executions/plan/8107798
- Failed case: `add-index-duraing-upgrade-cluster-dist-task`
- Case execution id: `19146628`
- Main log: https://tcms.pingcap.net/api/v1/artifact-files/ks3/logs/2026-03-22/plan-exec-8107798-re0/plan-exec-8107798-re0-case-4350009-2-2797896332/main.log
- Plan definition: `upgrade-with-ddlv8.4.0-v8.5.6-pre-no-dist-task@v8.5.6`
Effective parameters from the plan:
- source version: `v8.4.0`
- target version: `v8.5.6-pre`
- global variables:
- `tidb_ddl_enable_fast_reorg=0`
- `tidb_enable_dist_task=0`
### 2. Reproduction steps
The testcase logic is equivalent to:
1. Deploy a TiUP cluster with 3 TiDB + 3 TiKV on `v8.4.0`.
2. Set:
- `set global tidb_ddl_enable_fast_reorg = 0;`
- `set global tidb_enable_dist_task = 0;`
3. Restore the `sbtest1` database.
4. Concurrently run these DDLs:
- `alter table sbtest1.sbtest1 add index idxc (c, pad);`
- `alter table sbtest1.sbtest2 add index idxc (c, pad);`
- `alter table sbtest1.sbtest3 add index idxc (c, pad);`
- `alter table sbtest1.sbtest4 add index idxc (c, pad);`
- `alter table sbtest1.sbtest5 add index idxc (c, pad);`
- `alter table sbtest1.sbtest6 add index idxc (c, pad);`
5. Wait about 30 seconds while some add-index jobs are still running.
6. Upgrade the cluster from `v8.4.0` to `v8.5.6-pre`.
7. Wait until the DDL jobs become `synced`.
8. Run `admin check table` on the affected tables.
### 3. Expected behavior
After upgrade completes and all add-index jobs are `synced`, `admin check table` should pass.
### 4. Actual behavior
The testcase observed that all add-index jobs eventually became `synced`, but `admin check table` still failed with index/data inconsistency:
```text
Error 8223 (HY000): data inconsistency in table: sbtest1, index: idxc, handle: 428304, index-values:"" != record-values:"handle: 428304, values: [KindString 47286157983-07014258045-93418030894-13594793945-54305012376-15428050345-74044957252-16225186457-39760970876-03375109423 KindString 82325839586-17373146569-81799423616-83654270779-38218514799]"
```
Relevant log sequence from the testcase:
- upgrade completed successfully
- testcase logged `All DDL jobs are synced`
- then `admin check table sbtest1.sbtest1` failed with `Error 8223`
### 5. Extra observations from the testcase log
The testcase also saw transient connection errors during the rolling upgrade while the DDLs were still executing, for example:
- `unexpected EOF`
- `invalid connection`
However, those were not the final failure reason. The final failure was the post-upgrade consistency check after the jobs had already reached `synced`.
### 6. Environment / build info from testcase log
The upgraded cluster reported:
- Release Version: `v8.5.6`
- Git Commit Hash: `9707812e00107a41145208d91c4fafc2d6cefd4d`
- UTC Build Time: `2026-03-19 12:55:50`
### 7. Source code path of the testcase
For reference, the corresponding endless testcase is:
- `testcase/upgrade/enable_ddl/enable_ddl_test.go`
- `It("should success while there are some add index ddl running", ...)`
The testcase waits for all jobs to become `synced`, and then runs `admin check table`, which is where the inconsistency is detected.
Contributor guide
Assessment
This issue has not been assessed yet.