cockroachdb / cockroachdb/cockroach
roachtest: improve handling of cluster creation errors from cloud provider
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We have been seeing a number of cluster creation errors every night on roachtest nightly runs. Part of this has to do with GCP not having enough resources on `us-east1-b`, where we create VMs by default [1]. These errors typically look like:
```
The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
resources available to fulfill the request. '(resource type:compute)'.
```
Recent example: https://github.com/cockroachdb/cockroach/issues/108629#issuecomment-1811529312.
The error message includes a computer-readable payload that indicates other AZs where there _are_ resources available for the request that failed; that information should be in `errorInfo.metadatas.zonesAvailable`:
```
- errorInfo:
domain: compute.googleapis.com
metadatas:
attachment: local-ssd:16
vmType: n2-highcpu-96
zone: us-east1-b
zonesAvailable: us-east1-c,us-east1-d
reason: resource_availability
```
(extracted from the error message linked above).
Roachtest could be smarter about its cluster creation retry mechanism and take this information into account.
It also wouldn't hurt to rotate the default AZ (i.e., use `us-east1-{b,c,d}`), or even use `us-central` as well.
[1] https://github.com/cockroachdb/cockroach/blob/bab4335ba1c1d948f08c1e993a7407b23c6d4021/pkg/roachprod/vm/gce/gcloud.go#L969
Jira issue: CRDB-33544
Contributor guide
Assessment
This issue has not been assessed yet.