cockroachdb / cockroachdb/cockroach

roachtest: add detection for live host migration (GCE)

Open
#136,783 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement E-quick-win T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

### Background

Live host migrations are fairly common in GCE [1]. Seemingly, AWS _just_ rolled them out [2], but only for dedicated instances? To my knowledge, Azure doesn't have it either. Thus, at this time, we're only interested in the _detection_ of a live host migration in GCE.

By default, the framework sets `ClusterSpec.TerminateOnMigration` to `false`, meaning that GCE live host migration is allowed to migrate any host of that cluster, when it deems necessary. A small subset of roachtests (e.g., `clock/monotonic/xxx`) set it to `true`, in which case, a VM that must be migrated, will instead be _terminated_.

In order to improve our observability and reduce infra. noise, we need to detect a live migration, and its outcome. When `ClusterSpec.TerminateOnMigration == true`, and at least one VM in the cluster was terminated _due to_ migration, and the corresponding test fails, this should be treated as an infra. flake, thereby bypassing GH issue creation. When, `ClusterSpec.TerminateOnMigration == false`, and at least one VM in the cluster was migrated, and the corresponding test fails, then we should attach the following parameter label to the GH issue description `liveHostMigrations: n1,n3`; i.e., the value is a (sorted) list of node ids, which were migrated.

While many tests may not be impacted by the latency of live host migration, we should exclude _performance_ tests (to reduce variance between runs); i.e., for `TestSpec.Benchmark == true`, override `ClusterSpec.TerminateOnMigration = false`.

### Approach

GCE doesn't expose a direct callback/hook for host maintenance events. Instead, we have to query (or poll) the stackdriver log, similarly to `getPreemptedVMNames` or `getHostErrorVMNames`. The actual logic to read the log is similar to spotvm preemption detection [3].

**NOTE**: We should consider consolidating all three checks under the same polling logic (see `monitorForPreemptedVMs`). While polling may not be strictly needed for low frequency events like "host error" or "host migration", there isn't really a down-side; it's only a handful of extra API calls per (test runner) worker.

[1] https://cloud.google.com/compute/docs/instances/host-maintenance-overview#live_migrate
[2] https://aws.amazon.com/about-aws/whats-new/2024/10/amazon-ec2-dedicated-hosts-live-migration-based-host-maintenance/
[3] https://github.com/cockroachdb/cockroach/blob/a23be6bce928c3e08074a15815b3c67a657bb40e/pkg/roachprod/vm/gce/gcloud.go#L516

Jira issue: CRDB-45245

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.