github / github/gh-ost

FR Speed up the binary log application

Open
#454 2 comments 0 reactions 1 assignee Claimed by @shlomi-noach View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

I noticed that gh-ost runtime is sensitive to its physical location. For example, if you use --test-on-replica, the gh-ost executable needs to be located on the slave for best performance. On a WAN, it can make a 1 to 100 difference. It can be the reason gh-ost will never catch up and the migration will last forever.

The reason is simple :

when looking at the general log, there are a lot of repeated statements like this on the same table :

```

259010 Prepare replace /* gh-ost `ghost`.`_t_gho` */ into
`_ghost`.`_t_gho`
values
(?,, ?, ?)

259010 Execute replace /* gh-ost `ghost`.`t_gho` */ into ...
259010 Close stmt
```

A single event created 3 network round trips.
This can obviously be optimized.

If 2 consecutive events are hitting the same table and are of the same kind (insert, update), then they can be added at the end of the replace, up to a certain number in mysqldump fashion. The most efficient would be not to use prepared statement (that could be an option) and just generate a large statement ( 1 network round trip) or execute several statements per prepare if the next event is compatible ...

Using bulk statements or repeated execute would tremendously improve performance if the table is being written to when doing the schema change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.