cockroachdb / cockroachdb/cockroach

roachtest: enforce region affinity

Open
#121,437 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-foundations C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Roachtests can _optionally_ specify regions (and availability zones) via `ClusterSpec`, e.g., `ClusterSpec.GCE.Zones`. If the regions are specified, they will be used by roachprod, when provisioning a corresponding cluster. The specified regions are assumed to be _absolute_. That is, only availability zones are allowed to change during provisioning, while the regions never change. E.g., a transient cluster provisioning error may be retried in a different availability zone [1]. The primary reason for disallowing specified regions to change is _egress_. A number of roachtests may end up importing/exporting data from _regional_ cloud buckets [2].

When regions are _unspecified_ (via `ClusterSpec`), a cloud-specific default is chosen. The defaults correspond to the regional buckets (in each cloud), used for import/export. However, a default may be at odds with the availability of other resources [3], [4]. (E.g., GCE `t2a` instances are not available in `us-east1`.) Subsequently, switching to another region may result in unwanted _egress_, for the corresponding roachtest. Thus, we should consider how best to enforce "region affinity" in this case. Perhaps, this could be a heuristic based on the size and the type of a roachtest; e.g., large backup/restore tests should stay within the region defaults, unless otherwise specified (via `ClusterSpec`). Since input/output buckets aren't part of the spec., inferring those will be challenging. Making regions a required spec., for every roachtest seems rather inflexible.

[1] https://github.com/cockroachdb/cockroach/pull/120714
[2] https://github.com/cockroachdb/cockroach/issues/111371
[3] https://github.com/cockroachdb/cockroach/pull/117661
[4] https://github.com/cockroachdb/cockroach/issues/114523

Jira issue: CRDB-37257

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.