pingcap / pingcap/tidb

[Upgrade] Partial index predicate is silently lost when add-index DDL is queued as Job V1

Open
#70,281 8 comments 0 reactions 0 assignees View on GitHub
affects-8.5 component/ddl impact/upgrade severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

1. Start a TiDB v7.5.6 cluster and create the test data:

```sql
CREATE DATABASE pi_upgrade_repro;
CREATE TABLE pi_upgrade_repro.t (
id BIGINT PRIMARY KEY,
a INT,
b INT
);
INSERT INTO pi_upgrade_repro.t VALUES
(1, -1, 10), (2, 0, 20), (3, 1, 30), (4, 2, 40);
```

2. Start a standard rolling upgrade from v7.5.6 to v8.5.7. Upgrade PD and TiKV first, then keep at least one v7.5.6 TiDB and one v8.5.7 TiDB online. Put the cluster in the DDL upgrade state with `/upgrade/start`.

3. Confirm that the v8.5.7 TiDB is using DDL Job V1 because the TiDB registry still contains a pre-v8.4 TiDB. Enable fast reorg (`tidb_ddl_enable_fast_reorg=ON`).

4. Connect directly to the v8.5.7 TiDB and submit:

```sql
CREATE INDEX idx_b_partial
ON pi_upgrade_repro.t(b) WHERE a > 0;
```

The job is system-paused during the upgrade. In the reproduced run, the persisted job was `Version:v1`, had six V1 raw arguments, and had no persisted `ConditionString`.

5. Finish rolling all TiDB instances to v8.5.7, wait until every TiDB has switched its DDL job version to V2, and call `/upgrade/finish`. Let the original paused job resume; do not resubmit the SQL.

6. Inspect the resulting index:

```sql
SHOW CREATE TABLE pi_upgrade_repro.t;

SELECT INDEX_NAME, PREDICATE
FROM INFORMATION_SCHEMA.TIDB_INDEXES
WHERE TABLE_SCHEMA = 'pi_upgrade_repro'
AND TABLE_NAME = 't'
AND INDEX_NAME = 'idx_b_partial';

ADMIN CHECK TABLE pi_upgrade_repro.t;
ADMIN CHECK INDEX pi_upgrade_repro.t idx_b_partial;
```

This was reproduced independently in a TCMS all-target rolling-upgrade lifecycle and in a local TiUP Playground. A Job V2 negative control using the same table, data, and predicate created the correct partial index.

### 2. What did you expect to see? (Required)

The original predicate must survive the upgrade pause/resume lifecycle and the completed index must remain:

```sql
KEY `idx_b_partial` (`b`) WHERE `a` > 0
```

Only the two rows satisfying `a > 0` should have index entries. If Job V1 cannot represent the predicate, TiDB should reject the DDL with an actionable error instead of accepting it and changing its semantics.

### 3. What did you see instead (Required)

The DDL returned success without an error or warning, but the predicate was silently lost. The completed object was:

```sql
KEY `idx_b_partial` (`b`)
```

`INFORMATION_SCHEMA.TIDB_INDEXES.PREDICATE` was `NULL`, and all four rows had committed index entries rather than only the two rows satisfying `a > 0`.

Both `ADMIN CHECK TABLE` and `ADMIN CHECK INDEX` succeeded. They cannot detect this issue because the table and index are internally consistent as an ordinary full index; the failure is a silent change of the user's requested DDL semantics.

Relevant log/evidence sequence from the reproduction:

```text
jobVersion=v1
ID:180, State:pausing, Version:v1, ArgLen:6, query="... WHERE a > 0"
change job version in use: old=v1, new=v2
jobID=180, Version:v1, partial_condition_expr_string=""
table scan count=4, totalKeys=4, writtenKeys=4
State=synced, RowCount=4, Version:v1
```

The Job V2 negative control preserved `` `a` > 0 `` and wrote two index entries.

### 4. What is your TiDB version? (Required)

Source TiDB:

```text
Release Version: v7.5.6
Git Commit Hash: 1ad553d6de2cb0c9b2f3cfdb760aaae109b2c466
```

Target TiDB:

```text
Release Version: v8.5.7
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
```

Target PD and TiKV were also v8.5.7. TCMS evidence: https://tcms.pingcap.net/dashboard/executions/plan/8176782

## Compatibility Addendum

### Affected Version Matrix

- low version: TiDB v7.5.6
- high version: TiDB v8.5.7
- upgrade path: standard component-by-component rolling upgrade; PD and TiKV are upgraded before the TiDB rolling-upgrade phase
- feature window: partial index support is introduced into the release-8.5 line in v8.5.7; relevant schema/DDL backport: https://github.com/pingcap/tidb/pull/68831 (master PR: https://github.com/pingcap/tidb/pull/62759)

### Mixed-Version Actor Split

- old actor: a registered v7.5.6 TiDB, which causes the target submitter to keep using DDL Job V1
- new actor: the v8.5.7 TiDB that parses and submits the partial-index DDL, and later resumes the same job after all nodes are upgraded
- shared object or contract: persisted `ModifyIndexArgs` for an add-index DDL job, especially `IndexArg.ConditionString`

### Minimal Mixed Deployment Repro

- topology: target v8.5.7 PD/TiKV, plus mixed v7.5.6 and v8.5.7 TiDB instances during the TiDB rolling-upgrade phase
- trigger: submit `CREATE INDEX ... WHERE ...` to the v8.5.7 TiDB while it is still using Job V1; allow the upgrade gate to pause the job; finish the rollout and resume the same job

### Expected Behavior

Preserve the partial-index predicate end-to-end, or fail explicitly before persisting an unrepresentable Job V1.

### Actual Behavior

The predicate is absent from the persisted V1 arguments. The same job later completes successfully as a full index after the cluster is entirely on v8.5.7.

### Impact

- upgrade impact: a user can submit a valid new-version DDL to a new TiDB during the normal TiDB mixed-version window and receive a successful result with silently changed semantics
- rollback impact: not required to trigger this bug; the reproduction finishes on all-target v8.5.7
- user-visible effect: predicate rows and non-predicate rows are all indexed; storage/write amplification and the scope of uniqueness constraints can differ from what the user requested; ordinary ADMIN checks still pass

### Evidence

- TCMS standard all-target lifecycle: execution 8176782
- independent local TiUP Playground reproduction with the same v7.5.6/v8.5.7 actor split
- positive signal: original V1 job finishes as a four-entry full index with no predicate
- negative control: Job V2 retains the predicate and creates only two entries

### Suspected Root Cause

The v8.5.7 submit path normalizes the predicate into `IndexArg.ConditionString`, but `ModifyIndexArgs.getArgsV1()` serializes only the legacy add-index arguments and omits `ConditionString`. The upgrade gate marks the job system-paused but still fills and persists its arguments while the process is using Job V1. When the same job is resumed after the rollout, `decodeAddIndexV1()` yields an empty condition, so the v8.5.7 worker consistently builds an ordinary full index.

Suggested guard: reject partial add-index DDL when `job.Version == JobVersion1`, or provide a backward-compatible encoding that preserves the predicate. Add a regression covering `V1 submit + upgrade system pause + all-target V2 readiness + resume same job`.

Contributor guide

Open the contributing guide

Research direction

Start with ModifyIndexArgs.getArgsV1() and decodeAddIndexV1(), then trace how IndexArg.ConditionString is persisted when a V1 job is system-paused during upgrade. Add the regression described in the issue for V1 submission, pause, rollout, and resume; done means the predicate survives or the DDL fails explicitly instead of creating a full index.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.