pingcap / pingcap/tidb

DDL owner epoch token can collide across rapid owner handoff

Open
#69,755 1 comment 0 reactions 0 assignees View on GitHub
component/ddl found-by-ai severity/moderate sig/sql-infra type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

This is a deterministic unit-level reproduction for the DDL owner-epoch identity token. It has not yet been lifted to a full SQL/cluster chaos reproduction.

1. In the DDL owner scheduler, simulate the same TiDB instance retiring from DDL owner and then becoming owner again within the same wall-clock second.
2. Keep a stale reorg worker result from the previous owner epoch with `res.ownerTS` equal to the first owner epoch token.
3. Let the new owner epoch call the same result filter used by `runReorgJob`.
4. Observe that the previous owner epoch token and current owner epoch token can both be the same second-level Unix timestamp.

The current-source RED used this exact boundary:

```text
previousOwnerTS = 1000
curOwnerTS = 1000
```

Because `runReorgJob` treats `res.ownerTS == curTS` as proof that a reorg result belongs to the current owner epoch, a stale result from the previous owner epoch can pass the filter when the two owner epochs happen in the same second.

### 2. What did you expect to see? (Required)

Two distinct DDL owner epochs on the same TiDB should never share the same identity token. A late reorg result from a previous owner epoch should take the owner-handoff retry path instead of being accepted as a current-owner result.

### 3. What did you see instead (Required)

The owner token is minted from `time.Now().Unix()`. If the same TiDB retires and becomes DDL owner again within one wall-clock second, the two owner epochs can share the same `ownerTS` value.

This makes the async result filter too weak: equality of `ownerTS` no longer proves that the reorg result belongs to the current owner epoch.

The practical risk is around long DDL reorgs such as `ADD INDEX`: progress, row count, warnings, or a reorg error produced by a previous owner epoch may be accepted by a later owner epoch instead of being rejected and retried after owner handoff.

### 4. What is your TiDB version? (Required)

```text
Current master source commit: 13282a8bd06bd33324a4dbfd3c1c03685f3cd9aa
```

Likely root cause and fix direction

`OnBecomeOwner` uses a second-level wall-clock value as an owner identity token. `runReorgJob` later uses equality of that token to decide whether a reorg worker result belongs to the current owner epoch. A wall-clock second is not unique enough for an owner-epoch identity token.

A safer fix is to mint the token monotonically, for example:

```text
ownerTS = max(time.Now().Unix(), previousOwnerTS + 1)
```

The local GREEN used that shape: two owner epochs in the same wall-clock second received distinct tokens, so stale previous-owner results would no longer pass the equality filter.

Contributor guide

Open the contributing guide

Research direction

Locate OnBecomeOwner and runReorgJob in the DDL owner scheduler, then trace how ownerTS is created and compared with reorg results. Reproduce consecutive owner epochs with both timestamps set to 1000, and verify that the resulting tokens differ and stale results take the owner-handoff retry path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.