cockroachdb / cockroachdb/cockroach

roachtest: improve handling of cluster creation errors from cloud provider

Open
#114,523 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We have been seeing a number of cluster creation errors every night on roachtest nightly runs. Part of this has to do with GCP not having enough resources on `us-east1-b`, where we create VMs by default [1]. These errors typically look like:

```
The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
resources available to fulfill the request. '(resource type:compute)'.
```

Recent example: https://github.com/cockroachdb/cockroach/issues/108629#issuecomment-1811529312.

The error message includes a computer-readable payload that indicates other AZs where there _are_ resources available for the request that failed; that information should be in `errorInfo.metadatas.zonesAvailable`:

```
- errorInfo:
domain: compute.googleapis.com
metadatas:
attachment: local-ssd:16
vmType: n2-highcpu-96
zone: us-east1-b
zonesAvailable: us-east1-c,us-east1-d
reason: resource_availability
```

(extracted from the error message linked above).

Roachtest could be smarter about its cluster creation retry mechanism and take this information into account.

It also wouldn't hurt to rotate the default AZ (i.e., use `us-east1-{b,c,d}`), or even use `us-central` as well.

[1] https://github.com/cockroachdb/cockroach/blob/bab4335ba1c1d948f08c1e993a7407b23c6d4021/pkg/roachprod/vm/gce/gcloud.go#L969

Jira issue: CRDB-33544

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.