ddl stucks and keeps printing "syncer check all versions, someone is not synced" even though the job have already synced
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
timeline:
- older owner node A start schedule DDL, but hasn't started query the table
- another node B become new owner, and some client insert a create-table DDL job J
- node B move J to done state and start waiting all node synced
- node A query the job J out, found the job is in Done state and have un-synced MDL info, so it tries to sync it
- node B finish the sync, deleted all MDL keys of the job and finish the job J
- node A calls waitSchemaSyncedForMDL, it never ends as the MDL keys will not be write again, and it keeps printing the log
the reason is in older version TiDB, DDL scheduler loop doesn't cancel and exit, it keeps running even after it's not the owner anymore. it's detects whether it's the new owner inside the loop, so once it starts schedule jobs it won't return until it finished one job step
https://github.com/pingcap/tidb/blob/a7df4f9845d5d6a590c5d45dad0dcc9f21aa8765/pkg/ddl/job_table.go#L279-L284
marked as moderate as it's a corner case that's hard to reproduce in real world. I found it during upgrade to current master where we will force new node to be the owner, so there will be a owner change and have more chance to trigger this bug.
since v8.2.0, we have refactor this part to cancel the scheduler loop on retire owner, so no such issue
https://github.com/pingcap/tidb/blob/821e491a20fbab36604b36b647b5bae26a2c1418/pkg/ddl/job_table.go#L176-L177
### 2. What did you expect to see? (Required)
no stuck
### 3. What did you see instead (Required)
ddl stucks and keeps printing "syncer check all versions, someone is not synced"
### 4. What is your TiDB version? (Required)
Contributor guide
Assessment
This issue has not been assessed yet.