Direct rolling upgrade from v6.1.0 to v8.5.6 can get stuck when audit/plugin bootstrap DDL runs
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal Reproduce Step
Run a TiDB Cloud Dedicated rolling upgrade directly from `v6.1.0` to `v8.5.6` with:
- source version: `v6.1.0`
- target version: `v8.5.6`
- at least two TiDB instances
- audit/plugin loaded; the reproduced environment had `instance.plugin_load = audit-1`
- old and new TiDB nodes alive in the same mixed-version rolling-upgrade window
Reproduced TCMS execution:
- TPCC direct upgrade: https://tcms.pingcap.net/dashboard/executions/plan/8141748
This direct upgrade stayed running for about 3 hours without converging and was eventually cancelled.
### 2. What Did You Expect to See?
The direct rolling upgrade should either:
- complete successfully, with all TiDB nodes becoming Ready on `v8.5.6`; or
- fail fast with a clear diagnostic explaining that this direct mixed-version path is unsupported when audit/plugin bootstrap DDL is involved.
### 3. What Did You See Instead?
The direct rolling upgrade can stop in the TiDB rolling phase:
- the new TiDB process starts, but the SQL port does not become Ready;
- audit system-table DDL remains unfinished;
- the new TiDB keeps waiting for the old TiDB node to synchronize the schema version;
- the rolling upgrade does not advance to later SLA/configuration checks.
Observed stuck audit objects include:
- `mysql.audit_log_filters`
- `mysql.audit_log_filter_rules`
### 4. What Is Your TiDB Version?
Source:
```text
TiDB v6.1.0
commit: 1a89decdb192cbdce6a7b0020d71128bc964d30f
bootstrap version: version91
```
Target:
```text
TiDB v8.5.6
commit: ae18096e023780bb56bfce33698abec0d4640d0a
bootstrap version: version227
```
### 5. Log Evidence
Log window: `2026-05-19 15:00:00 CST` to `2026-05-19 18:20:00 CST`.
TCMS execution: `8141748`
Cluster ID: `10692412739140410681`
Status: `CANCELLED`
The v6.1.0 TiDB process was started with audit plugin enabled:
```text
--plugin-load audit-1
```
At `2026-05-19 16:27:48 CST`, the new TiDB `v8.5.6` process started and loaded the audit plugin:
```text
[INFO] [printer.go:52] ["Welcome to TiDB."] ["Release Version"=v8.5.6]
[INFO] [plugin.go:377] ["start load plugin manifest"] [plugin=audit-1] [name=audit] [version=1]
```
At `2026-05-19 16:28:51 CST`, the audit plugin submitted system-table DDL:
```text
["DDL job submitted"] [category=ddl] [job="ID:164, Type:create table, State:queueing, SchemaState:none, SchemaID:3, TableID:163, ..."] [query="CREATE TABLE IF NOT EXISTS mysql.audit_log_filters ..."]
```
At the end of the selected window, `2026-05-19 18:19:59 CST`, DDL schema synchronization was still waiting for the old TiDB node:
```text
["syncer check all versions, someone is not synced"] [category=ddl] [info="instance ip db-tidb-0.db-tidb-peer.tidb10692412739140410681.svc, port 4000, ..."] ["ddl job id"=164] [ver=168]
```
### 6. Analysis / Suspected Cause
This appears to be a mixed-version compatibility issue across the v6.2 concurrent-DDL framework boundary.
Shared state involved:
- DDL job queue/state
- DDL owner
- schema version synchronization
- audit/plugin system-table metadata
Actor split in the rolling-upgrade window:
- old actor: `v6.1.0` TiDB remains live, may be the DDL owner, and participates in schema-version synchronization;
- new actor: `v8.5.6` TiDB starts, runs plugin/bootstrap system DDL, and waits for cluster-wide schema synchronization.
Suspected failure shape:
1. The `v8.5.6` TiDB starts audit/plugin bootstrap DDL, such as creating or checking `mysql.audit_log_filters` / `mysql.audit_log_filter_rules`.
2. The DDL/sync path crosses the legacy v6.1 DDL queue and the newer concurrent-DDL/table-queue behavior introduced after v6.2.
3. The new TiDB waits for the old TiDB to synchronize the schema version.
4. The old TiDB does not complete the schema-sync contract expected by the new version.
5. The new TiDB remains SQL-not-Ready and the rolling upgrade stalls.
### 7. Workaround
For clusters that load audit/plugin/extension code which may produce system DDL during upgrade, avoid the direct `v6.1.0 -> v8.5.6` rolling-upgrade path.
Use a staged upgrade path instead, for example:
```text
v6.1.0 -> v6.5.12 -> v7.5.7 -> v8.5.6
```
Also freeze user/tool-driven DDL, ADD INDEX, IMPORT INTO, LOAD DATA, BR restore, and DM/TiCDC schema changes during each upgrade window.
Contributor guide
Research direction
Start by reproducing the v6.1.0-to-v8.5.6 mixed-version upgrade with the audit plugin enabled, then inspect the DDL bootstrap and schema-version synchronization paths described in the report. Focus on the audit_log_filters and audit_log_filter_rules DDL and the waiting old TiDB node. Done means the upgrade converges or fails fast with a clear unsupported-path diagnostic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100