github / github/gh-ost

FR/RFC: optional flag for GTID cleanup after test/migrate on replica

Open
#254 23 comments 1 reaction 0 assignees View on GitHub
enhancement suggested for community contribution
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

As Shlomi mentioned in his recent blog post, GTID is problematic with `--test-on-replica` or `--migrate-on-replica` because the replica will no longer be promotable due to having extra gtid_executed transactions. As alluded to in the post, there are workarounds for this. On the GTID side of things, I can think of a couple different ways:

(a) Insert empty transactions with correct gtid_next on the master (and replicating down to all replicas), so that gtid_executed of all nodes now effectively matches the modified replica's
or
(b) Adjust the modified replica's gtid_executed via convoluted RESET MASTER song-and-dance

Both solutions require SUPER. Hopefully there are RDS equivalents but this needs more research.

If there's interest, I might be able to spend some time on a PR in a few months (just not in Oct, hoping to release another project before the Percona Live submission deadline :) Any guidance on which solution is more palatable?

Option (a) is a bit cumbersome in the `--migrate-on-replica` case, since it may require a very large number of empty transactions to be inserted, and it involves interacting with the master. OTOH, this could be implemented as a general-purpose gtid_executed-syncing external tool which gh-ost just shells out to at the appropriate time.

Option (b) limits its impact to just the modified replica. The drawbacks are that all binlogs on the replica get deleted (probably ok since they're polluted with all those extra writes anyway); and the modified replica must not have replicas of its own. The automated procedure is something like this:

BEFORE MIGRATION:
- On the target replica, parse the chunk of gtid_executed corresponding to the replica's own server_uuid (if any) and store in a variable. This will let us figure out what to "restore" to in order to effectively undo the gtids assigned via gh-ost writes.

AFTER MIGRATION:
- Use `PURGE BINARY LOGS TO ''` to delete all old binlogs. This is just done here first so that the later `RESET MASTER` step is fast, in cases where there are lots of large binlogs and filesystems where deletion is slow
- `STOP SLAVE` -- so that we can get a consistent position. The steps after are fast so replication won't need to be stopped for more than a split second.
- Query @@global.gtid_executed and store in a variable, and then modify it so that the chunk corresponding to the replica's own server_uuid matches its prior state. i.e., figure out what its gtid_executed "should" be if the gh-ost transactions had not occurred.
- `RESET MASTER` -- this will nuke gtid_executed and gtid_purged, along with the one remaining current binlog
- `SET GLOBAL GTID_PURGED = ''` -- this also inherently sets gtid_executed
- `FLUSH LOCAL BINARY LOGS` -- roll to a new binlog immediately, so that the gtid_executed value gets persisted in 5.6, which weirdly uses the binlogs as only persisted source of truth
- `PURGE BINARY LOGS TO ''` -- erase the first binlog file in case binlog_gtid_simple_recovery is in use, since it would cause issues upon mysqld restart
- `START SLAVE` (unless leaving this as the user's responsibility, I think that's already the assumption in `--test-on-replica`)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.