github / github/gh-ost

lock hang at cut-over table stage

Open
#939 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

We met a lock hang at cut-over table stage recently. After analysis, I have a suspicion that it is related to the PR #888. The problem occurs handle `Timeout while waiting for events up to lock` in `atomicCutOver()`. After timeout error current atomicCutOver will be cancelled and try again. When canceled, defer func is executed, include `okToUnlockTable <-true` and `this.applier.DropAtomicCutOverSentryTableIfExists()`, also `applier.AtomicCutOverMagicLock` will drop magic cut-over table after recv okToUnlockTable channel. so the PR use sync.Once to avoid to send drop cutover sentry table to mysql twice, But if the drop table operation executed by `applier.DropAtomicCutOverSentryTableIfExists()` first, it will be locked with `Waiting for table metadata lock`, while the actual lock owner `applier.AtomicCutOverMagicLock` is stuck here (Once mutex lock), Waiting for the completion of the former.

This can be reproduced by injecting some problems,at [here](https://github.com/github/gh-ost/blob/master/go/logic/migrator.go#L638) force a timeout error before waiting for the event to be locked, and Wait a few seconds [here](https://github.com/github/gh-ost/blob/master/go/logic/applier.go#L871) to make sure that the delete table is invoking by DropAtomicCutOverSentryTableIfExists().

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.