cockroachdb / cockroachdb/cockroach
disaster-recovery: Backup failure due to error from GCS as Received 429 `"Too many requests."`
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The backup failure is because GCS could not handle the number of request at that specific point in time. This is because if the rate-limiting on the GCS side. But, in this particular scenario, there can be an exponential retry as the rate may eventually come down.
Job reference - https://drt-chaos-0002.drt.crdb.io:26258/#/jobs/989603761182736390.
Error:
```
unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroachdb-backup-testing/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachprod-scheduled-backups%2Fdrt-chaos%2Fsystem%2F1714752607933605000%2Fmetadata%2Flatest&prettyPrint=false&projection=full&startOffset=&versions=false": compute: Received 429 `"Too many requests."`
```
Jira issue: CRDB-40657
Contributor guide
Assessment
This issue has not been assessed yet.