github / github/gh-ost

ERROR Timeout while waiting for events up to lock.

Open
#700 4 comments 0 reactions 0 assignees View on GitHub
duplicate
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

command:
gh-ost --allow-on-master --assume-rbr --initially-drop-ghost-table --initially-drop-old-table --chunk-size=1500 --hooks-path=/tmp/hook --exact-rowcount --user=root--password="123.com" --host=10.10.1.202 --port=3306 --assume-master-host=10.10.1.202:3306 --alter='xxx' -execute

The error:
```shell
2019-01-23 11:27:44 INFO Looking for magic cut-over table.

2019-01-23 11:27:44 INFO Will now proceed to drop magic table and unlock tables.

2019-01-23 11:27:44 INFO Dropping magic cut-over table.

2019-01-23 11:27:44 INFO Releasing lock from `broker`.`b_invoice`, `broker`.`_b_invoice_del`.

2019-01-23 11:27:45 INFO Tables unlocked.

2019-01-23 11:27:45 DEBUG Getting nothing in the write queue. Sleeping....

Copy: 24339/24339 100.0%; Applied: 0; Backlog: 0/1000; Time: 28s(total), 7s(copy); streamer: mysql-bin.005986:88395447; State: migrating; ETA: due.

2019-01-23 11:27:46 DEBUG Getting nothing in the write queue. Sleeping....

Copy: 24339/24339 100.0%; Applied: 0; Backlog: 0/1000; Time: 29s(total), 7s(copy); streamer: mysql-bin.005986:88551253; State: migrating; ETA: due.

2019-01-23 11:27:46 DEBUG checking for cut-over postpone.

2019-01-23 11:27:46 DEBUG checking for cut-over postpone: complete.

2019-01-23 11:27:46 INFO Grabbing voluntary lock: gh-ost.8877.lock.

2019-01-23 11:27:46 INFO Setting LOCK timeout as 6 seconds.

2019-01-23 11:27:46 INFO Looking for magic cut-over table.

2019-01-23 11:27:46 INFO Creating magic cut-over table `broker`.`_b_invoice_del`.

2019-01-23 11:27:46 INFO Magic cut-over table created.

2019-01-23 11:27:46 INFO Locking `broker`.`b_invoice`, `broker`.`_b_invoice_del`.

2019-01-23 11:27:47 INFO Tables locked.

2019-01-23 11:27:47 INFO Session locking original & magic tables is 8877.

2019-01-23 11:27:47 INFO Writing changelog state: AllEventsUpToLockProcessed:1548214067024736192.

2019-01-23 11:27:47 INFO Waiting for events up to lock.

2019-01-23 11:27:47 DEBUG Getting nothing in the write queue. Sleeping....

Copy: 24339/24339 100.0%; Applied: 0; Backlog: 0/1000; Time: 30s(total), 7s(copy); streamer: mysql-bin.005986:88671411; State: migrating; ETA: due.

2019-01-23 11:27:48 DEBUG Getting nothing in the write queue. Sleeping....

Copy: 24339/24339 100.0%; Applied: 0; Backlog: 0/1000; Time: 31s(total), 7s(copy); streamer: mysql-bin.005986:88958928; State: migrating; ETA: due.

2019-01-23 11:27:49 DEBUG Getting nothing in the write queue. Sleeping....

Copy: 24339/24339 100.0%; Applied: 0; Backlog: 0/1000; Time: 32s(total), 7s(copy); streamer: mysql-bin.005986:89309509; State: migrating; ETA: due.

2019-01-23 11:27:50 ERROR Timeout while waiting for events up to lock.

2019-01-23 11:27:50 ERROR 2019-01-23 11:27:50 ERROR Timeout while waiting for events up to lock.

goroutine 1 [running]:.

runtime/debug.Stack(0x5f, 0x100, 0xc42001e1e0).

/usr/local/go/src/runtime/debug/stack.go:24 +0xa7.

runtime/debug.PrintStack().

/usr/local/go/src/runtime/debug/stack.go:16 +0x22.

github.com/github/gh-ost/vendor/github.com/outbrain/golib/log.logErrorEntry(0x2, 0x93ed00, 0xc4204dcf00, 0x0, 0x100000001).

/home/shlomi-noach/go/src/github.com/github/gh-ost/vendor/github.com/outbrain/golib/log/log.go:178 +0xfd.

github.com/github/gh-ost/vendor/github.com/outbrain/golib/log.Errore(0x93ed00, 0xc4204dcf00, 0xc4204dcf00, 0x1).

/home/shlomi-noach/go/src/github.com/github/gh-ost/vendor/github.com/outbrain/golib/log/log.go:224 +0x3e.

github.com/github/gh-ost/go/logic.(*Migrator).atomicCutOver(0xc420122090, 0x0, 0x0).

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/logic/migrator.go:634 +0xa48.

github.com/github/gh-ost/go/logic.(*Migrator).cutOver(0xc420122090, 0x2, 0xc420240b80).

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/logic/migrator.go:524 +0x286.

github.com/github/gh-ost/go/logic.(*Migrator).(github.com/github/gh-ost/go/logic.cutOver)-fm(0x3b9aca00, 0xc420192620).

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/logic/migrator.go:413 +0x2a.

github.com/github/gh-ost/go/logic.(*Migrator).retryOperation(0xc420122090, 0xc4202905d0, 0x0, 0x0, 0x0, 0x1, 0xc4202905d0).

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/logic/migrator.go:140 +0x75.

github.com/github/gh-ost/go/logic.(*Migrator).(github.com/github/gh-ost/go/logic.retryOperation)-fm(0xc4202905d0, 0x0, 0x0, 0x0, 0x1, 0xc4200181b0).

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/logic/migrator.go:411 +0x52.

github.com/github/gh-ost/go/logic.(*Migrator).Migrate(0xc420122090, 0x0, 0x0).

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/logic/migrator.go:413 +0x873.

main.main().

/home/shlomi-noach/go/src/github.com/github/gh-ost/go/cmd/gh-ost/main.go:252 +0x1da0.
```

What is the cause? please, thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.