pingcap / pingcap/tidb

autoid: AUTO_ID_CACHE=1 allocator can issue duplicate IDs during cross-database RENAME TABLE

Open
#70,784 6 comments 0 reactions 1 assignee Claimed by @YangKeao View on GitHub
may-affects-25.10 may-affects-26.3 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major sig/sql-infra type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This issue affects the single-point auto-increment allocator used by tables created with `AUTO_ID_CACHE=1`.

During a cross-database `RENAME TABLE`, allocator transfer depends on each TiDB node applying the incremental InfoSchema change and running `alloc.Transfer(newDBID, tableID)`. If one TiDB node is delayed immediately before `Transfer`, while another TiDB node has already loaded the renamed table and uses the destination DB ID, the same physical table ID can allocate from two independent autoid-service keys:

- source key: `(oldDBID, tableID)`
- destination key: `(newDBID, tableID)`

The steps below use a small test-only failpoint to make this window deterministic. The failpoint is only for reproducing the issue; it is not a proposed production fix.

#### 1. Apply this test-only failpoint patch

```diff
diff --git a/pkg/infoschema/builder.go b/pkg/infoschema/builder.go
index 52038756de..946a7ed70b 100644
--- a/pkg/infoschema/builder.go
+++ b/pkg/infoschema/builder.go
@@ -869,6 +869,7 @@ func applyCreateTable(b *Builder, m meta.Reader, dbInfo *model.DBInfo, tableID i
ConvertOldVersionUTF8ToUTF8MB4IfNeed(tblInfo)

for _, alloc := range allocs.Allocs {
+ failpoint.Inject("delayBeforeTransferAllocator", func() {})
err := alloc.Transfer(dbInfo.ID, tableID)
if err != nil {
return nil, errors.Trace(err)
```

`pkg/infoschema/builder.go` already imports `github.com/pingcap/failpoint`, so no import change is needed.

#### 2. Build a failpoint-enabled TiDB server

```bash
git fetch upstream
git checkout upstream/master
make server_failpoint TARGET=/tmp/autoid-sp-transfer-repro/tidb-server-fp
```

#### 3. Start a local 1-PD / 1-TiKV / 2-TiDB cluster

Use any free ports. The following commands use:

- PD: `127.0.0.1:22379`
- TiKV: `127.0.0.1:22160`
- old-transaction TiDB: SQL `127.0.0.1:24000`, status `127.0.0.1:24080`
- rename/new-table TiDB: SQL `127.0.0.1:24001`, status `127.0.0.1:24081`

```bash
RUN=/tmp/autoid-sp-transfer-repro
mkdir -p "$RUN"/pd "$RUN"/tikv "$RUN"/logs

pd-server \
--name=pd-fp \
--data-dir="$RUN/pd" \
--client-urls=http://127.0.0.1:22379 \
--peer-urls=http://127.0.0.1:22380 \
--advertise-client-urls=http://127.0.0.1:22379 \
--advertise-peer-urls=http://127.0.0.1:22380 \
--initial-cluster=pd-fp=http://127.0.0.1:22380 \
--log-file="$RUN/logs/pd.log" \
--force-new-cluster &

tikv-server \
--pd-endpoints=127.0.0.1:22379 \
--addr=127.0.0.1:22160 \
--advertise-addr=127.0.0.1:22160 \
--status-addr=127.0.0.1:22180 \
--data-dir="$RUN/tikv" \
--log-file="$RUN/logs/tikv.log" &

GO_FAILPOINTS='github.com/pingcap/tidb/pkg/server/enableTestAPI=return' \
"$RUN/tidb-server-fp" \
--store=tikv \
--path=127.0.0.1:22379 \
--host=127.0.0.1 \
-P=24000 \
--status=24080 \
--advertise-address=127.0.0.1 \
--lease=1s \
--log-file="$RUN/logs/tidb-1.log" &

GO_FAILPOINTS='github.com/pingcap/tidb/pkg/server/enableTestAPI=return' \
"$RUN/tidb-server-fp" \
--store=tikv \
--path=127.0.0.1:22379 \
--host=127.0.0.1 \
-P=24001 \
--status=24081 \
--advertise-address=127.0.0.1 \
--lease=1s \
--log-file="$RUN/logs/tidb-2.log" &
```

Check both TiDB nodes are up and metadata lock is enabled:

```bash
mysql -h127.0.0.1 -P24000 -uroot -e 'select version(), @@port, @@global.tidb_enable_metadata_lock'
mysql -h127.0.0.1 -P24001 -uroot -e 'select version(), @@port, @@global.tidb_enable_metadata_lock'
```

Expected output includes `@@global.tidb_enable_metadata_lock = 1`.

#### 4. Run this external client program

```bash
mkdir -p /tmp/autoid-sp-client-repro
cd /tmp/autoid-sp-client-repro

cat > go.mod <<'EOF'
module autoid-sp-client-repro

go 1.23

require github.com/go-sql-driver/mysql v1.9.3
EOF

cat > main.go <<'EOF'
package main

import (
"bytes"
"context"
"database/sql"
"fmt"
"log"
"net/http"
"time"

_ "github.com/go-sql-driver/mysql"
)

const (
oldAddr = "127.0.0.1:24000"
newAddr = "127.0.0.1:24001"
oldStatusAddr = "127.0.0.1:24080"
failpointPath = "github.com/pingcap/tidb/pkg/infoschema/delayBeforeTransferAllocator"
failpointExpr = "sleep(5000)"
oldSchema = "sp_old_repro"
newSchema = "sp_new_repro"
waitNewVisible = 10 * time.Second
)

func main() {
ctx := context.Background()
oldDB := mustOpen(oldAddr)
defer oldDB.Close()
newDB := mustOpen(newAddr)
defer newDB.Close()

logServer(ctx, "old", oldDB)
logServer(ctx, "new", newDB)

cleanup(ctx, newDB)
defer cleanup(ctx, newDB)

mustExec(ctx, newDB, "CREATE DATABASE "+oldSchema)
mustExec(ctx, newDB, "CREATE DATABASE "+newSchema)
mustExec(ctx, newDB, fmt.Sprintf(
"CREATE TABLE %s.t (id BIGINT PRIMARY KEY AUTO_INCREMENT, v BIGINT) /*T![auto_id_cache] AUTO_ID_CACHE=1 */",
oldSchema,
))

oldConn, err := oldDB.Conn(ctx)
if err != nil {
log.Fatal(err)
}
defer oldConn.Close()

mustConnExec(ctx, oldConn, "BEGIN OPTIMISTIC")
mustConnQueryClose(ctx, oldConn, fmt.Sprintf("SELECT * FROM %s.t WHERE id = -1", oldSchema))

mustSetFailpoint(ctx)

ddlDone := make(chan error, 1)
go func() {
_, err := newDB.ExecContext(ctx, fmt.Sprintf("RENAME TABLE %s.t TO %s.t", oldSchema, newSchema))
ddlDone <- err
}()

if !waitForNewTable(ctx, newDB) {
mustDeleteFailpoint(context.Background())
_, _ = oldConn.ExecContext(ctx, "ROLLBACK")
log.Fatal("new table did not become visible")
}

newID := mustInsert(ctx, newDB, fmt.Sprintf("INSERT INTO %s.t(v) VALUES (200)", newSchema))
oldID := mustConnInsert(ctx, oldConn, fmt.Sprintf("INSERT INTO %s.t(v) VALUES (100)", oldSchema))

mustDeleteFailpoint(context.Background())
_, _ = oldConn.ExecContext(ctx, "ROLLBACK")

if err := waitDDL(ddlDone, 20*time.Second); err != nil {
log.Fatalf("rename DDL did not finish: %v", err)
}

log.Printf("old_id=%d new_id=%d", oldID, newID)
if oldID == newID {
log.Printf("REPRODUCED: duplicate auto id allocated: old_id=%d new_id=%d", oldID, newID)
return
}
}

func mustOpen(addr string) *sql.DB {
db, err := sql.Open("mysql", fmt.Sprintf("root:@tcp(%s)/?interpolateParams=true&timeout=3s&readTimeout=30s&writeTimeout=30s", addr))
if err != nil {
log.Fatal(err)
}
return db
}

func logServer(ctx context.Context, label string, db *sql.DB) {
var version, port, mdl string
err := db.QueryRowContext(ctx, "SELECT VERSION(), @@port, @@global.tidb_enable_metadata_lock").Scan(&version, &port, &mdl)
if err != nil {
log.Fatal(err)
}
log.Printf("%s server version=%s port=%s metadata_lock=%s", label, version, port, mdl)
}

func mustExec(ctx context.Context, db *sql.DB, sqlText string) {
if _, err := db.ExecContext(ctx, sqlText); err != nil {
log.Fatalf("%s: %v", sqlText, err)
}
}

func mustConnExec(ctx context.Context, conn *sql.Conn, sqlText string) {
if _, err := conn.ExecContext(ctx, sqlText); err != nil {
log.Fatalf("%s: %v", sqlText, err)
}
}

func mustConnQueryClose(ctx context.Context, conn *sql.Conn, sqlText string) {
rows, err := conn.QueryContext(ctx, sqlText)
if err != nil {
log.Fatalf("%s: %v", sqlText, err)
}
_ = rows.Close()
}

func mustInsert(ctx context.Context, db *sql.DB, sqlText string) int64 {
r, err := db.ExecContext(ctx, sqlText)
if err != nil {
log.Fatalf("%s: %v", sqlText, err)
}
id, err := r.LastInsertId()
if err != nil {
log.Fatal(err)
}
return id
}

func mustConnInsert(ctx context.Context, conn *sql.Conn, sqlText string) int64 {
r, err := conn.ExecContext(ctx, sqlText)
if err != nil {
log.Fatalf("%s: %v", sqlText, err)
}
id, err := r.LastInsertId()
if err != nil {
log.Fatal(err)
}
return id
}

func waitForNewTable(ctx context.Context, db *sql.DB) bool {
deadline := time.Now().Add(waitNewVisible)
for time.Now().Before(deadline) {
rows, err := db.QueryContext(ctx, fmt.Sprintf("SELECT 1 FROM %s.t LIMIT 0", newSchema))
if err == nil {
_ = rows.Close()
return true
}
time.Sleep(10 * time.Millisecond)
}
return false
}

func waitDDL(ch <-chan error, timeout time.Duration) error {
select {
case err := <-ch:
return err
case <-time.After(timeout):
return fmt.Errorf("timed out")
}
}

func cleanup(ctx context.Context, db *sql.DB) {
_, _ = db.ExecContext(ctx, "DROP DATABASE IF EXISTS "+oldSchema)
_, _ = db.ExecContext(ctx, "DROP DATABASE IF EXISTS "+newSchema)
}

func mustSetFailpoint(ctx context.Context) {
url := fmt.Sprintf("http://%s/fail/%s", oldStatusAddr, failpointPath)
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewBufferString(failpointExpr))
if err != nil {
log.Fatal(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
log.Fatalf("set failpoint got %s", resp.Status)
}
log.Printf("enabled failpoint %s on %s with %q", failpointPath, oldStatusAddr, failpointExpr)
}

func mustDeleteFailpoint(ctx context.Context) {
url := fmt.Sprintf("http://%s/fail/%s", oldStatusAddr, failpointPath)
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, url, nil)
if err != nil {
log.Fatal(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
log.Fatalf("delete failpoint got %s", resp.Status)
}
log.Printf("disabled failpoint %s on %s", failpointPath, oldStatusAddr)
}
EOF

go mod tidy
go run .
```

Observed output from my local reproduction:

```text
old server version=8.0.11-TiDB-v9.0.0-beta.2.pre-2176-gdb35d47066-dirty port=24000 metadata_lock=1
new server version=8.0.11-TiDB-v9.0.0-beta.2.pre-2176-gdb35d47066-dirty port=24001 metadata_lock=1
enabled failpoint github.com/pingcap/tidb/pkg/infoschema/delayBeforeTransferAllocator on 127.0.0.1:24080 with "sleep(5000)"
disabled failpoint github.com/pingcap/tidb/pkg/infoschema/delayBeforeTransferAllocator on 127.0.0.1:24080
old_id=1 new_id=1
REPRODUCED: duplicate auto id allocated: old_id=1 new_id=1
```

The old transaction is rolled back at the end only so that the DDL can finish and the test can clean up. The allocator bug is already visible before rollback: both successful `INSERT` statements returned the same auto-generated ID.

Useful log evidence from the old TiDB node:

```text
[2026/09/01 16:03:24.984 +08:00] [INFO] [job_worker.go:855] ["run one job step"] [jobID=142] [job="ID:142, Type:rename table, ... SchemaID:138, TableID:140 ..."]
[2026/09/01 16:03:25.000 +08:00] [INFO] [autoid.go:194] ["alloc4Signed from"] [dbID=138] [tblID=140] ["from base"=0] ["from end"=0] ["to base"=0] ["to end"=4000]
[2026/09/01 16:03:25.004 +08:00] [INFO] [autoid.go:194] ["alloc4Signed from"] [dbID=136] [tblID=140] ["from base"=0] ["from end"=0] ["to base"=0] ["to end"=4000]
[2026/09/01 16:03:29.995 +08:00] [INFO] [loader.go:216] ["diff load InfoSchema success"] [currVer=77] [neededVer=78] [gotVer=78] ["elapsed time"=5.000915371s] [phyTblIDs="[140]"] [actionTypes="[14]"] [diffTypes="[\"rename table\"]"]
```

The two `alloc4Signed` lines show that the same physical table ID (`tblID=140`) got ranges from two different DB IDs (`dbID=138` and `dbID=136`) during the cross-database rename.

### 2. What did you expect to see? (Required)

For a table using `AUTO_ID_CACHE=1`, TiDB should not allocate the same auto-increment ID for the same physical table during or after `RENAME TABLE old_db.t TO new_db.t`.

Even if one TiDB node is slow to apply the rename InfoSchema diff, stale sessions using the old table object should not be able to allocate from the old `(oldDBID, tableID)` single-point allocator while another node has already started allocating from the destination `(newDBID, tableID)` allocator.

Possible acceptable behaviors include:

- all allocations are serialized through one authoritative key;
- stale old-table allocations are blocked/retried until the allocator has been transferred;
- the transferred allocator keeps using the old DB ID when the table metadata says the auto ID schema should remain the old schema.

### 3. What did you see instead (Required)

Two successful `INSERT` statements returned the same generated auto-increment ID:

```text
old_id=1 new_id=1
```

This happened with `tidb_enable_metadata_lock=1`.

The failure requires an artificially delayed old TiDB node to make the race deterministic, but the underlying invariant violation is that the single-point autoid service key is based on `(dbID, tableID)`, while cross-database rename can temporarily allow one TiDB node/session to use the old DB ID and another TiDB node to use the new DB ID for the same physical table ID.

### 4. What is your TiDB version? (Required)

Built from current `upstream/master` plus only the test failpoint patch shown above.

```text
Release Version: v9.0.0-beta.2.pre-2176-gdb35d47066-dirty
Edition: Community
Git Commit Hash: db35d47066648fe73abce6318d53fc625df51490
Git Branch: review/autoid-sp-transfer-rename
UTC Build Time: 2026-09-01 07:56:23
GoVersion: go1.26.5-X:nodwarf5
Race Enabled: false
Check Table Before Drop: false
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.