cockroachdb / cockroachdb/cockroach

disaster-recovery: Backup failure due to error from GCS as Received 429 `"Too many requests."`

Open
#127,895 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-master C-bug O-testcluster P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The backup failure is because GCS could not handle the number of request at that specific point in time. This is because if the rate-limiting on the GCS side. But, in this particular scenario, there can be an exponential retry as the rate may eventually come down.

Job reference - https://drt-chaos-0002.drt.crdb.io:26258/#/jobs/989603761182736390.

Error:
```
unable to list files in gcs bucket: Get "https://storage.googleapis.com/storage/v1/b/cockroachdb-backup-testing/o?alt=json&delimiter=&endOffset=&includeTrailingDelimiter=false&pageToken=&prefix=roachprod-scheduled-backups%2Fdrt-chaos%2Fsystem%2F1714752607933605000%2Fmetadata%2Flatest&prettyPrint=false&projection=full&startOffset=&versions=false": compute: Received 429 `"Too many requests."`
```

Jira issue: CRDB-40657

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.