github / github/gh-ost

concurrent migration issue

Open
#439 15 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

Gh-ost version : 1.0.30

`**gh-ost was designed with having multiple concurrent migration running in parallel** in triggerless-design.md`

BUT I fond gh-ost could not concurrent migration actually , one task wolud be quit with error "ERROR Timeout while waiting for events up to lock" , another would be OK .

execute script:
Task 1.
```
gh-ost --host=10.28.77.43 --database=ghostdb --table=test1g --port=55944 --alter="add column addcol05 int default 0" --concurrent-rowcount --conf=/usr/local/gh-ost.cnf --throttle-flag-file="/tmp/gh-ost.ghostdb.test1g.throttle" --postpone-cut-over-flag-file="/tmp/gh-ost.ghostdb.test1g.postpone" --hooks-path="/tmp" --max-load="Threads_connected=5000,Threads_running=5000" --initially-drop-old-table --initially-drop-ghost-table --initially-drop-socket-file --verbose --execute
```

Task 2.
```
gh-ost --host=10.28.77.43 --database=ghostdb --table=test10g --port=55944 --alter="add column addcol05 int default 0" --concurrent-rowcount --conf=/usr/local/gh-ost.cnf --throttle-flag-file="/tmp/gh-ost.ghostdb.test10g.throttle" --postpone-cut-over-flag-file="/tmp/gh-ost.ghostdb.test10g.postpone" --hooks-path="/tmp" --max-load="Threads_connected=5000,Threads_running=5000" --initially-drop-old-table --initially-drop-ghost-table --initially-drop-socket-file --verbose --execute
```

Task 1's log
```
2017-07-04 14:50:33 INFO Looking for magic cut-over table
2017-07-04 14:50:33 INFO Will now proceed to drop magic table and unlock tables
2017-07-04 14:50:33 INFO Dropping magic cut-over table
2017-07-04 14:50:33 INFO Releasing lock from `ghostdb`.`test1g`, `ghostdb`.`_test1g_del`
2017-07-04 14:50:33 INFO Tables unlocked
2017-07-04 14:50:34 INFO Grabbing voluntary lock: gh-ost.2447063.lock
2017-07-04 14:50:34 INFO Setting LOCK timeout as 6 seconds
2017-07-04 14:50:34 INFO Looking for magic cut-over table
2017-07-04 14:50:34 INFO Creating magic cut-over table `ghostdb`.`_test1g_del`
2017-07-04 14:50:34 INFO Magic cut-over table created
2017-07-04 14:50:34 INFO Locking `ghostdb`.`test1g`, `ghostdb`.`_test1g_del`
2017-07-04 14:50:34 INFO Tables locked
2017-07-04 14:50:34 INFO Session locking original & magic tables is 2447063
2017-07-04 14:50:34 INFO Writing changelog state: AllEventsUpToLockProcessed:1499151034706034765
2017-07-04 14:50:34 INFO Waiting for events up to lock
2017-07-04 14:50:35 INFO StreamEvents encountered unexpected error: ERROR 1236 (HY000): A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'SVR14883DE630-bin.001315' at 4, the last event read from './SVR14883DE630-bin.001315' at 187061838, the last byte read from './SVR14883DE630-bin.001315' at 187061838.
Copy: 4378153/4378153 100.0%; Applied: 0; Backlog: 0/100; Time: 6m10s(total), 2m9s(copy); streamer: SVR14883DE630-bin.001315:187061838; State: migrating; ETA: due
2017-07-04 14:50:37 ERROR Timeout while waiting for events up to lock
2017-07-04 14:50:37 ERROR 2017-07-04 14:50:37 ERROR Timeout while waiting for events up to lock
2017-07-04 14:50:37 INFO Looking for magic cut-over table
2017-07-04 14:50:37 INFO Will now proceed to drop magic table and unlock tables
2017-07-04 14:50:37 INFO Dropping magic cut-over table
2017-07-04 14:50:37 INFO Removing socket file: /tmp/gh-ost.ghostdb.test1g.sock
2017-07-04 14:50:37 INFO executing gh-ost-on-failure hook: /tmp/gh-ost-on-failure.hook
**2017-07-04 14:50:37 FATAL 2017-07-04 14:50:37 ERROR Timeout while waiting for events up to lock**
```

Task 2's log

```
2017-07-04 14:55:15 INFO Waiting for events up to lock: got AllEventsUpToLockProcessed:1499151314500641336
2017-07-04 14:55:15 INFO Done waiting for events up to lock; duration=993.440857ms
Copy: 18898698/18898698 100.0%; Applied: 0; Backlog: 0/100; Time: 10m53s(total), 10m50s(copy); streamer: SVR14883DE630-bin.001323:668568927; State: migrating; ETA: due
2017-07-04 14:55:15 INFO Setting RENAME timeout as 3 seconds
2017-07-04 14:55:15 INFO Session renaming tables is 2447290
2017-07-04 14:55:15 INFO Issuing and expecting this to block: rename /* gh-ost */ table `ghostdb`.`test10g` to `ghostdb`.`_test10g_del`, `ghostdb`.`_test10g_gho` to `ghostdb`.`test10g`
2017-07-04 14:55:15 INFO Found atomic RENAME to be blocking, as expected. Double checking the lock is still in place (though I don't strictly have to)
2017-07-04 14:55:15 INFO Checking session lock: gh-ost.2446665.lock
2017-07-04 14:55:15 INFO Connection holding lock on original table still exists
2017-07-04 14:55:15 INFO Will now proceed to drop magic table and unlock tables
2017-07-04 14:55:15 INFO Dropping magic cut-over table
2017-07-04 14:55:15 INFO Releasing lock from `ghostdb`.`test10g`, `ghostdb`.`_test10g_del`
2017-07-04 14:55:15 INFO Tables unlocked
2017-07-04 14:55:15 INFO Tables renamed
2017-07-04 14:55:15 INFO Lock & rename duration: 1.041417629s. During this time, queries on `test10g` were blocked
2017-07-04 14:55:15 INFO Looking for magic cut-over table
2017-07-04 14:55:16 INFO Closed streamer connection. err=
2017-07-04 14:55:16 INFO Droppping table `ghostdb`.`_test10g_ghc`
2017-07-04 14:55:16 INFO StreamEvents encountered unexpected error: sync is been closing...
2017-07-04 14:55:16 INFO Table dropped
2017-07-04 14:55:16 INFO Am not dropping old table because I want this operation to be as live as possible. If you insist I should do it, please add `--ok-to-drop-table` next time. But I prefer you do not. To drop the old table, issue:
2017-07-04 14:55:16 INFO -- drop table `ghostdb`.`_test10g_del`
2017-07-04 14:55:16 INFO executing gh-ost-on-success hook: /tmp/gh-ost-on-success.hook

```

BTW , gh-ost-on-failure hook can not work normally in this condition.

`2017-07-04 14:50:37 INFO executing gh-ost-on-failure hook: /tmp/gh-ost-on-failure.hook`

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.