Question: Primary instance restarts
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
## Context
We're running a larger migration of a data type int -> bigint on a table (1.4bn rows / 2TB size).
Running on a CloudSQL MySQL 5.7 primary (`--allow-on-master` with one replica). Replica has parallel replication enabled.
The instance is now scheduled for maintenance by google.
### Command we're running
```
gh-ost --max-load=Threads_running=25 \
--critical-load=Threads_running=300 \
--chunk-size=1000 \
--throttle-control-replicas=replica_ip \
--user="ghost-replica" \
--password=xxx \
--host=primary_ip \
--allow-on-master \
--database="integrations" \
--table="sales" \
--verbose \
--throttle-flag-file="/tmp/ghost-throttle-pos" \
--postpone-cut-over-flag-file="/tmp/ghost-postpone-pos" \
--panic-flag-file="/tmp/ghost-panic-pos" \
--alter="MODIFY COLUMN id bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT" \
--execute
```
## Problem
Initial testing of this scenario showed that if a migration was running showed that if the primary restarted, gh-ost would not reconnect and show errors about its host connection going away, and would no longer migrate. The program never exited, it would just sit with no more updates occuring.
## Questions
* Is this avoidable?
* Can I change something in my interactive shell that would re-start the connection to the host?
* Is the cutover dependent on replica lag? Is a viable workaround (to speed this up) to ignore our replica lag, and let the copy "run wild" with no consideration of replica lag, reducing our days to completion? Knowing that will mean a multi-day replication lag on the replica that we can tolerate for a short time.
While we can postpone the maintenance window of the database, we are doubtful the migration will finish in time. ~45% complete, ETA ~30 days to complete.
In the mean time we are trying any optimization to reduce writes to our DB, to relieve the bin log and hope the migration is throttled less.
Thanks in advance!
Contributor guide
Assessment
This issue has not been assessed yet.