pingcap / pingcap/tidb

add index during upgrade from v8.4.0 to v8.5.6-pre may leave index/data inconsistency after jobs are synced

Open
#67,250 5 comments 0 reactions 0 assignees View on GitHub
affects-8.5 component/ddl contribution severity/critical type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce path from TCMS

This issue comes from TCMS plan execution `8107798`:
- Plan execution: https://tcms.pingcap.net/dashboard/executions/plan/8107798
- Failed case: `add-index-duraing-upgrade-cluster-dist-task`
- Case execution id: `19146628`
- Main log: https://tcms.pingcap.net/api/v1/artifact-files/ks3/logs/2026-03-22/plan-exec-8107798-re0/plan-exec-8107798-re0-case-4350009-2-2797896332/main.log
- Plan definition: `upgrade-with-ddlv8.4.0-v8.5.6-pre-no-dist-task@v8.5.6`

Effective parameters from the plan:
- source version: `v8.4.0`
- target version: `v8.5.6-pre`
- global variables:
- `tidb_ddl_enable_fast_reorg=0`
- `tidb_enable_dist_task=0`

### 2. Reproduction steps

The testcase logic is equivalent to:

1. Deploy a TiUP cluster with 3 TiDB + 3 TiKV on `v8.4.0`.
2. Set:
- `set global tidb_ddl_enable_fast_reorg = 0;`
- `set global tidb_enable_dist_task = 0;`
3. Restore the `sbtest1` database.
4. Concurrently run these DDLs:
- `alter table sbtest1.sbtest1 add index idxc (c, pad);`
- `alter table sbtest1.sbtest2 add index idxc (c, pad);`
- `alter table sbtest1.sbtest3 add index idxc (c, pad);`
- `alter table sbtest1.sbtest4 add index idxc (c, pad);`
- `alter table sbtest1.sbtest5 add index idxc (c, pad);`
- `alter table sbtest1.sbtest6 add index idxc (c, pad);`
5. Wait about 30 seconds while some add-index jobs are still running.
6. Upgrade the cluster from `v8.4.0` to `v8.5.6-pre`.
7. Wait until the DDL jobs become `synced`.
8. Run `admin check table` on the affected tables.

### 3. Expected behavior

After upgrade completes and all add-index jobs are `synced`, `admin check table` should pass.

### 4. Actual behavior

The testcase observed that all add-index jobs eventually became `synced`, but `admin check table` still failed with index/data inconsistency:

```text
Error 8223 (HY000): data inconsistency in table: sbtest1, index: idxc, handle: 428304, index-values:"" != record-values:"handle: 428304, values: [KindString 47286157983-07014258045-93418030894-13594793945-54305012376-15428050345-74044957252-16225186457-39760970876-03375109423 KindString 82325839586-17373146569-81799423616-83654270779-38218514799]"
```

Relevant log sequence from the testcase:
- upgrade completed successfully
- testcase logged `All DDL jobs are synced`
- then `admin check table sbtest1.sbtest1` failed with `Error 8223`

### 5. Extra observations from the testcase log

The testcase also saw transient connection errors during the rolling upgrade while the DDLs were still executing, for example:
- `unexpected EOF`
- `invalid connection`

However, those were not the final failure reason. The final failure was the post-upgrade consistency check after the jobs had already reached `synced`.

### 6. Environment / build info from testcase log

The upgraded cluster reported:
- Release Version: `v8.5.6`
- Git Commit Hash: `9707812e00107a41145208d91c4fafc2d6cefd4d`
- UTC Build Time: `2026-03-19 12:55:50`

### 7. Source code path of the testcase

For reference, the corresponding endless testcase is:
- `testcase/upgrade/enable_ddl/enable_ddl_test.go`
- `It("should success while there are some add index ddl running", ...)`

The testcase waits for all jobs to become `synced`, and then runs `admin check table`, which is where the inconsistency is detected.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.