github / github/gh-ost

lots of metalock ,mysql crash

Open
#768 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

### report issue

``` gh-ost: version ,1.0.46, ```

```mysql:version,5.6.28```

#### issue:

​ ``` Thanks for gh-ost you guys provide, we have do five hundred thousand ddl ```

​ ```Recently ,we got some strange questions ,some times gh-ost ddl crash our master db. (lots of meta lock in db , thread running increase suddenly ,then database was crash)```

``` 1. db load was very low,little tps/qps```

```2. lots of meta lock in processlist snapshot.```

​ ```We set rename timeout is 1 seconds (cut-over-lock-timeout-seconds=1). ```

```The key point here,gh-ost seems didn't release the lock after lock wait time exceeded ```

##### DDL LOG : we could found gh-ost didn't print log about "release the lock"

```Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 14s(total), 12s(copy); streamer: mysql-bin.000021:256866991; State: migrating; ETA: due
2019-07-23 22:26:16 INFO Setting RENAME timeout as 1 seconds
2019-07-23 22:26:16 INFO Session renaming tables is 3277189
2019-07-23 22:26:16 INFO Issuing and expecting this to block: rename table `magi`.`trial_lucky` to `magi`.`_trial_lucky_1563891962_del`, `magi`.`_trial_lucky_1563891962_gho` to `magi`.`trial_lucky`
2019-07-23 22:26:17 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 15s(total), 12s(copy); streamer: mysql-bin.000021:256867677; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 16s(total), 12s(copy); streamer: mysql-bin.000021:256868164; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 17s(total), 12s(copy); streamer: mysql-bin.000021:256868651; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 18s(total), 12s(copy); streamer: mysql-bin.000021:256869138; State: migrating; ETA: due
Copy: 152422/152422 100.0%; Applied: 2; Backlog: 0/1000; Time: 19s(total), 12s(copy); streamer: mysql-bin.000021:256869625; State: migrating; ETA: due
[mysql] 2019/07/23 22:27:53 driver.go:81: net.Error from Dial()': dial tcp 172.21.80.17:3306: connect: connection timed out
[mysql] 2019/07/23 22:28:22 packets.go:36: read tcp 172.20.101.133:18424->172.21.80.17:3306: read: connection reset by peer
[mysql] 2019/07/23 22:28:22 packets.go:36: read tcp 172.20.101.133:58982->172.21.80.17:3306: read: connection reset by peer
[mysql] 2019/07/23 22:28:22 packets.go:36: read tcp 172.20.101.133:51922->172.21.80.17:3306: read: connection reset by peer
``
```

##### DB PROCESSLIST SNAPSHOT: many metalock in process list

```'magi', 'Query', 1, 'Waiting for table metadata lock',
magi', 'Query', 1, 'Waiting for table metadata lock',
magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
magi', 'Query', 1, 'Waiting for table metadata lock',
'magi', 'Query', 1, 'Waiting for table metadata lock',
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.