DDL owner epoch token can collide across rapid owner handoff
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
This is a deterministic unit-level reproduction for the DDL owner-epoch identity token. It has not yet been lifted to a full SQL/cluster chaos reproduction.
1. In the DDL owner scheduler, simulate the same TiDB instance retiring from DDL owner and then becoming owner again within the same wall-clock second.
2. Keep a stale reorg worker result from the previous owner epoch with `res.ownerTS` equal to the first owner epoch token.
3. Let the new owner epoch call the same result filter used by `runReorgJob`.
4. Observe that the previous owner epoch token and current owner epoch token can both be the same second-level Unix timestamp.
The current-source RED used this exact boundary:
```text
previousOwnerTS = 1000
curOwnerTS = 1000
```
Because `runReorgJob` treats `res.ownerTS == curTS` as proof that a reorg result belongs to the current owner epoch, a stale result from the previous owner epoch can pass the filter when the two owner epochs happen in the same second.
### 2. What did you expect to see? (Required)
Two distinct DDL owner epochs on the same TiDB should never share the same identity token. A late reorg result from a previous owner epoch should take the owner-handoff retry path instead of being accepted as a current-owner result.
### 3. What did you see instead (Required)
The owner token is minted from `time.Now().Unix()`. If the same TiDB retires and becomes DDL owner again within one wall-clock second, the two owner epochs can share the same `ownerTS` value.
This makes the async result filter too weak: equality of `ownerTS` no longer proves that the reorg result belongs to the current owner epoch.
The practical risk is around long DDL reorgs such as `ADD INDEX`: progress, row count, warnings, or a reorg error produced by a previous owner epoch may be accepted by a later owner epoch instead of being rejected and retried after owner handoff.
### 4. What is your TiDB version? (Required)
```text
Current master source commit: 13282a8bd06bd33324a4dbfd3c1c03685f3cd9aa
```
Likely root cause and fix direction
`OnBecomeOwner` uses a second-level wall-clock value as an owner identity token. `runReorgJob` later uses equality of that token to decide whether a reorg worker result belongs to the current owner epoch. A wall-clock second is not unique enough for an owner-epoch identity token.
A safer fix is to mint the token monotonically, for example:
```text
ownerTS = max(time.Now().Unix(), previousOwnerTS + 1)
```
The local GREEN used that shape: two owner epochs in the same wall-clock second received distinct tokens, so stale previous-owner results would no longer pass the equality filter.
Contributor guide
Research direction
Locate OnBecomeOwner and runReorgJob in the DDL owner scheduler, then trace how ownerTS is created and compared with reorg results. Reproduce consecutive owner epochs with both timestamps set to 1000, and verify that the resulting tokens differ and stale results take the owner-handoff retry path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100