cockroachdb / cockroachdb/cockroach
backupccl: gs:// with AUTH=implicit hangs when metadata service unreachable
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
*Steps to Reproduce:*
1) while running cockroach as not-root on a GCP machine, drop non-root traffic to the instance metadata service:
`sudo iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP`
2) `SHOW BACKUPS IN 'gs://cockroach-fixtures?AUTH=implicit`
*Expected result:*
Some sort of error message saying that implicit auth failed
*Actual result:*
Hangs forever
Version tested: v22.2.2
Stack of stuck SHOW:
```
goroutine 9158 [select]:
github.com/googleapis/gax-go/v2.Sleep({0x61c6a68, 0xc008438330}, 0x7f6733f14108?)
github.com/googleapis/gax-go/v2/external/com_github_googleapis_gax_go_v2/invoke.go:57 +0x7b
cloud.google.com/go/internal.retry({0x61c6a68, 0xc008438330}, {0x3b9aca00, 0x6fc23ac00, 0x4000000000000000, 0x6fc23ac00}, 0xc003f13858, 0x561db10)
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:49 +0x14d
cloud.google.com/go/internal.Retry(...)
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:33
cloud.google.com/go/storage.run({0x61c6a68?, 0xc008438330?}, 0x52f8df2?, 0x9?, 0x0?)
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:54 +0xce
cloud.google.com/go/storage.(*ObjectIterator).fetch(0xc0082d9ae0, 0x0, {0x0, 0x0})
cloud.google.com/go/storage/external/com_google_cloud_go_storage/bucket.go:1558 +0xb7e
google.golang.org/api/iterator.(*PageInfo).fill(0xc00809b220, 0x40bde6f?)
google.golang.org/api/iterator/external/org_golang_google_api/iterator/iterator.go:139 +0x33
google.golang.org/api/iterator.(*PageInfo).next(0xc00809b220)
google.golang.org/api/iterator/external/org_golang_google_api/iterator/iterator.go:118 +0x9c
cloud.google.com/go/storage.(*ObjectIterator).Next(0xc0082d9ae0)
cloud.google.com/go/storage/external/com_google_cloud_go_storage/bucket.go:1525 +0x29
github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).List(0xc004cabdc0, {0x61c6a68, 0xc007b20060}, {0x0?, 0x4618e7?}, {0x52de95e, 0x5}, 0xc004a6d210)
github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:328 +0x246
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.ListFullBackupsInCollection({0x61c6a68, 0xc007b20060}, {0x6215b80, 0xc004cabe00})
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:483 +0xc3
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.showBackupsInCollectionPlanHook.func1({0x61c6a68?, 0xc007b20030?}, {0x61c6a68?, 0xc00a521b30?, 0xf0f000000000000?}, 0x61c6a68?)
github.com/cockroachdb/cockroach/pkg/ccl/backupccl/show.go:1337 +0x1d5
github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1()
github.com/cockroachdb/cockroach/pkg/sql/planhook.go:199 +0xa4
created by github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec
github.com/cockroachdb/cockroach/pkg/sql/planhook.go:197 +0x1b2
```
Jira issue: CRDB-24093
Contributor guide
Assessment
This issue has not been assessed yet.