cockroachdb / cockroachdb/cockroach
roachtest: enforce region affinity
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Roachtests can _optionally_ specify regions (and availability zones) via `ClusterSpec`, e.g., `ClusterSpec.GCE.Zones`. If the regions are specified, they will be used by roachprod, when provisioning a corresponding cluster. The specified regions are assumed to be _absolute_. That is, only availability zones are allowed to change during provisioning, while the regions never change. E.g., a transient cluster provisioning error may be retried in a different availability zone [1]. The primary reason for disallowing specified regions to change is _egress_. A number of roachtests may end up importing/exporting data from _regional_ cloud buckets [2].
When regions are _unspecified_ (via `ClusterSpec`), a cloud-specific default is chosen. The defaults correspond to the regional buckets (in each cloud), used for import/export. However, a default may be at odds with the availability of other resources [3], [4]. (E.g., GCE `t2a` instances are not available in `us-east1`.) Subsequently, switching to another region may result in unwanted _egress_, for the corresponding roachtest. Thus, we should consider how best to enforce "region affinity" in this case. Perhaps, this could be a heuristic based on the size and the type of a roachtest; e.g., large backup/restore tests should stay within the region defaults, unless otherwise specified (via `ClusterSpec`). Since input/output buckets aren't part of the spec., inferring those will be challenging. Making regions a required spec., for every roachtest seems rather inflexible.
[1] https://github.com/cockroachdb/cockroach/pull/120714
[2] https://github.com/cockroachdb/cockroach/issues/111371
[3] https://github.com/cockroachdb/cockroach/pull/117661
[4] https://github.com/cockroachdb/cockroach/issues/114523
Jira issue: CRDB-37257
Contributor guide
Assessment
This issue has not been assessed yet.