cockroachdb / cockroachdb/cockroach

roachprod: MIG quota and resource errors

Open
#120,827 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

With the addition of MIGs [1] the cluster creation can become stuck when waiting for the MIG to reach stability. Especially when the quota limit is reached or there are no resources available.

MIG scaling errors (creating new instances from the template) are not reported during creation. Currently, `roachprod` will wait for a MIG to become stable (`gcloud compute instance-groups managed wait-until --stable`) before proceeding [2].

At the moment there is no `timeout` specified for the command. A timeout should be added (could be based on a minimum and the number of instances requested).

Additionally, there could be a way to monitor for errors when the MIG is trying to allocate resources. This should be investigated as an additional option for better error reporting other than just a timeout.

[1] #118207
[2] https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/wait-until

Jira issue: CRDB-36889

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.