cockroachdb / cockroachdb/cockroach
roachtest: add detection for live host migration (Azure)
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Live VM migrations in Azure [1] appear to be much more disruptive than their analogue in GCE [2]. The delay causes the wall clock to get out of sync by more than 400ms, causing the correspond node to `panic`. Recently, we've seen a high rate of roachtest failures on ARM64 [3], due to the clock issue; as a temporary workaround, ARM64 runs were disabled. We've also seen AMD64 failures [4], although those appear at a much lower rate.
Note that unlike in GCE, there doesn't appear to be an option to shutdown a VM instead of a live migration; see the related issue for Cockroach Cloud [4].
[1] https://techcommunity.microsoft.com/blog/azurearchitectureblog/demystifying-azure-vm-maintenance-a-practical-guide-to-minimizing-disruptions/4077523
[2] https://github.com/cockroachdb/cockroach/issues/136783
[3] https://github.com/cockroachdb/cockroach/issues/138726
[4] https://github.com/cockroachdb/cockroach/issues/142969
[4] https://cockroachlabs.atlassian.net/browse/CC-31242
Jira issue: CRDB-48812
Contributor guide
Assessment
This issue has not been assessed yet.