cockroachdb / cockroachdb/cockroach
roachprod: MIG quota and resource errors
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
With the addition of MIGs [1] the cluster creation can become stuck when waiting for the MIG to reach stability. Especially when the quota limit is reached or there are no resources available.
MIG scaling errors (creating new instances from the template) are not reported during creation. Currently, `roachprod` will wait for a MIG to become stable (`gcloud compute instance-groups managed wait-until --stable`) before proceeding [2].
At the moment there is no `timeout` specified for the command. A timeout should be added (could be based on a minimum and the number of instances requested).
Additionally, there could be a way to monitor for errors when the MIG is trying to allocate resources. This should be investigated as an additional option for better error reporting other than just a timeout.
[1] #118207
[2] https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/wait-until
Jira issue: CRDB-36889
Contributor guide
Assessment
This issue has not been assessed yet.