cockroachdb / cockroachdb/cockroach
roachtest: add detection for live host migration (GCE)
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
### Background
Live host migrations are fairly common in GCE [1]. Seemingly, AWS _just_ rolled them out [2], but only for dedicated instances? To my knowledge, Azure doesn't have it either. Thus, at this time, we're only interested in the _detection_ of a live host migration in GCE.
By default, the framework sets `ClusterSpec.TerminateOnMigration` to `false`, meaning that GCE live host migration is allowed to migrate any host of that cluster, when it deems necessary. A small subset of roachtests (e.g., `clock/monotonic/xxx`) set it to `true`, in which case, a VM that must be migrated, will instead be _terminated_.
In order to improve our observability and reduce infra. noise, we need to detect a live migration, and its outcome. When `ClusterSpec.TerminateOnMigration == true`, and at least one VM in the cluster was terminated _due to_ migration, and the corresponding test fails, this should be treated as an infra. flake, thereby bypassing GH issue creation. When, `ClusterSpec.TerminateOnMigration == false`, and at least one VM in the cluster was migrated, and the corresponding test fails, then we should attach the following parameter label to the GH issue description `liveHostMigrations: n1,n3`; i.e., the value is a (sorted) list of node ids, which were migrated.
While many tests may not be impacted by the latency of live host migration, we should exclude _performance_ tests (to reduce variance between runs); i.e., for `TestSpec.Benchmark == true`, override `ClusterSpec.TerminateOnMigration = false`.
### Approach
GCE doesn't expose a direct callback/hook for host maintenance events. Instead, we have to query (or poll) the stackdriver log, similarly to `getPreemptedVMNames` or `getHostErrorVMNames`. The actual logic to read the log is similar to spotvm preemption detection [3].
**NOTE**: We should consider consolidating all three checks under the same polling logic (see `monitorForPreemptedVMs`). While polling may not be strictly needed for low frequency events like "host error" or "host migration", there isn't really a down-side; it's only a handful of extra API calls per (test runner) worker.
[1] https://cloud.google.com/compute/docs/instances/host-maintenance-overview#live_migrate
[2] https://aws.amazon.com/about-aws/whats-new/2024/10/amazon-ec2-dedicated-hosts-live-migration-based-host-maintenance/
[3] https://github.com/cockroachdb/cockroach/blob/a23be6bce928c3e08074a15815b3c67a657bb40e/pkg/roachprod/vm/gce/gcloud.go#L516
Jira issue: CRDB-45245
Contributor guide
Assessment
This issue has not been assessed yet.