lots of metalock ,mysql crash
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
### report issue
``` gh-ost: version ,1.0.46, ```
```mysql:version,5.6.28```
#### issue:
``` Thanks for gh-ost you guys provide, we have do five hundred thousand ddl ```
```Recently ,we got some strange questions ,some times gh-ost ddl crash our master db. (lots of meta lock in db , thread running increase suddenly ,then database was crash)```
``` 1. db load was very low,little tps/qps```
```2. lots of meta lock in processlist snapshot.```
```We set rename timeout is 1 seconds (cut-over-lock-timeout-seconds=1). ```
```The key point here,gh-ost seems didn't release the lock after lock wait time exceeded ```
##### DDL LOG : we could found gh-ost didn't print log about "release the lock"
```Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 14s(total), 12s(copy); streamer: mysql-bin.000021:256866991; State: migrating; ETA: due
2019-07-23 22:26:16 INFO Setting RENAME timeout as 1 seconds
2019-07-23 22:26:16 INFO Session renaming tables is 3277189
2019-07-23 22:26:16 INFO Issuing and expecting this to block: rename table `magi`.`trial_lucky` to `magi`.`_trial_lucky_1563891962_del`, `magi`.`_trial_lucky_1563891962_gho` to `magi`.`trial_lucky`
2019-07-23 22:26:17 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 15s(total), 12s(copy); streamer: mysql-bin.000021:256867677; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 16s(total), 12s(copy); streamer: mysql-bin.000021:256868164; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 17s(total), 12s(copy); streamer: mysql-bin.000021:256868651; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 18s(total), 12s(copy); streamer: mysql-bin.000021:256869138; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 19s(total), 12s(copy); streamer: mysql-bin.000021:256869625; State: migrating; ETA: due
[mysql] 2019/07/23 22:27:53 driver.go:81: net.Error from Dial()': dial tcp 172.21.80.17:3306: connect: connection timed out
[mysql] 2019/07/23 22:28:22 packets.go:36: read tcp 172.20.101.133:18424->172.21.80.17:3306: read: connection reset by peer
[mysql] 2019/07/23 22:28:22 packets.go:36: read tcp 172.20.101.133:58982->172.21.80.17:3306: read: connection reset by peer
[mysql] 2019/07/23 22:28:22 packets.go:36: read tcp 172.20.101.133:51922->172.21.80.17:3306: read: connection reset by peer
``
```
##### DB PROCESSLIST SNAPSHOT: many metalock in process list
```'magi', 'Query', 1, 'Waiting for table metadata lock',
magi', 'Query', 1, 'Waiting for table metadata lock',
magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
```
Contributor guide
Assessment
This issue has not been assessed yet.