lock hang at cut-over table stage
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
We met a lock hang at cut-over table stage recently. After analysis, I have a suspicion that it is related to the PR #888. The problem occurs handle `Timeout while waiting for events up to lock` in `atomicCutOver()`. After timeout error current atomicCutOver will be cancelled and try again. When canceled, defer func is executed, include `okToUnlockTable <-true` and `this.applier.DropAtomicCutOverSentryTableIfExists()`, also `applier.AtomicCutOverMagicLock` will drop magic cut-over table after recv okToUnlockTable channel. so the PR use sync.Once to avoid to send drop cutover sentry table to mysql twice, But if the drop table operation executed by `applier.DropAtomicCutOverSentryTableIfExists()` first, it will be locked with `Waiting for table metadata lock`, while the actual lock owner `applier.AtomicCutOverMagicLock` is stuck here (Once mutex lock), Waiting for the completion of the former.
This can be reproduced by injecting some problems,at [here](https://github.com/github/gh-ost/blob/master/go/logic/migrator.go#L638) force a timeout error before waiting for the event to be locked, and Wait a few seconds [here](https://github.com/github/gh-ost/blob/master/go/logic/applier.go#L871) to make sure that the delete table is invoking by DropAtomicCutOverSentryTableIfExists().
Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.