pingcap / pingcap/tidb

ddl stucks and keeps printing "syncer check all versions, someone is not synced" even though the job have already synced

Open
#57,003 3 comments 0 reactions 0 assignees View on GitHub
affects-6.5 affects-7.1 affects-7.5 affects-8.1 component/ddl severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

timeline:
- older owner node A start schedule DDL, but hasn't started query the table
- another node B become new owner, and some client insert a create-table DDL job J
- node B move J to done state and start waiting all node synced
- node A query the job J out, found the job is in Done state and have un-synced MDL info, so it tries to sync it
- node B finish the sync, deleted all MDL keys of the job and finish the job J
- node A calls waitSchemaSyncedForMDL, it never ends as the MDL keys will not be write again, and it keeps printing the log

the reason is in older version TiDB, DDL scheduler loop doesn't cancel and exit, it keeps running even after it's not the owner anymore. it's detects whether it's the new owner inside the loop, so once it starts schedule jobs it won't return until it finished one job step

https://github.com/pingcap/tidb/blob/a7df4f9845d5d6a590c5d45dad0dcc9f21aa8765/pkg/ddl/job_table.go#L279-L284

marked as moderate as it's a corner case that's hard to reproduce in real world. I found it during upgrade to current master where we will force new node to be the owner, so there will be a owner change and have more chance to trigger this bug.

since v8.2.0, we have refactor this part to cancel the scheduler loop on retire owner, so no such issue
https://github.com/pingcap/tidb/blob/821e491a20fbab36604b36b647b5bae26a2c1418/pkg/ddl/job_table.go#L176-L177

### 2. What did you expect to see? (Required)
no stuck
### 3. What did you see instead (Required)
ddl stucks and keeps printing "syncer check all versions, someone is not synced"
### 4. What is your TiDB version? (Required)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.