cockroachdb / cockroachdb/cockroach

roachtest: add detection for live host migration (Azure)

Open
#143,397 7 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement E-quick-win E-starter T-testeng target-release-25.3.0 v25.3.0-prerelease
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Live VM migrations in Azure [1] appear to be much more disruptive than their analogue in GCE [2]. The delay causes the wall clock to get out of sync by more than 400ms, causing the correspond node to `panic`. Recently, we've seen a high rate of roachtest failures on ARM64 [3], due to the clock issue; as a temporary workaround, ARM64 runs were disabled. We've also seen AMD64 failures [4], although those appear at a much lower rate.

Note that unlike in GCE, there doesn't appear to be an option to shutdown a VM instead of a live migration; see the related issue for Cockroach Cloud [4].

[1] https://techcommunity.microsoft.com/blog/azurearchitectureblog/demystifying-azure-vm-maintenance-a-practical-guide-to-minimizing-disruptions/4077523
[2] https://github.com/cockroachdb/cockroach/issues/136783
[3] https://github.com/cockroachdb/cockroach/issues/138726
[4] https://github.com/cockroachdb/cockroach/issues/142969
[4] https://cockroachlabs.atlassian.net/browse/CC-31242

Jira issue: CRDB-48812

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.