backupccl: v23.1.13: panic in authhandler

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/4791472886/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/4791472886/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:884
| runtime.panicmem
| GOROOT/src/runtime/panic.go:260
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:839
| golang.org/x/oauth2/authhandler.authHandlerSource.Token
| golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go:80
| golang.org/x/oauth2.(*reuseTokenSource).Token
| golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go:304
| golang.org/x/oauth2.(*Transport).RoundTrip
| golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go:45
| net/http.send
| GOROOT/src/net/http/client.go:251
| net/http.(*Client).send
| GOROOT/src/net/http/client.go:175
| net/http.(*Client).do
| GOROOT/src/net/http/client.go:715
| net/http.(*Client).Do
| GOROOT/src/net/http/client.go:581
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1
| cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:834
| cloud.google.com/go/storage.run.func1
| cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:67
| cloud.google.com/go/internal.retry
| cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:40
| cloud.google.com/go/internal.Retry
| cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:33
| cloud.google.com/go/storage.run
| cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:65
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2
| cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:833
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader
| cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:887
| cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader
| cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go:118
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFileAt.func1
| github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:291
| github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open.func1
| github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:227
| github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry.func1
| github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:110
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
| github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:195
| github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
| github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:172
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
| github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:194
| github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry
| github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:109
| github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open
| github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:224
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFileAt
| github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:300
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile
| github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:276
| github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).ReadFile
| github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:331
| github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.containsManifest
| github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:71
| github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.ResolveDest
| github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:169
| github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*backupResumer).Resume
| github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_job.go:503
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/ccl/backupccl/backup_job.go#L502-L504](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/ccl/backupccl/backup_job.go#L502-L504)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/ccl/backupccl/backupdest/backup_destination.go#L168-L170](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/ccl/backupccl/backupdest/backup_destination.go#L168-L170)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/ccl/backupccl/backupdest/backup_destination.go#L70-L72](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/ccl/backupccl/backupdest/backup_destination.go#L70-L72)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/impl_registry.go#L330-L332](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/impl_registry.go#L330-L332)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/gcp/gcs_storage.go#L275-L277](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/gcp/gcs_storage.go#L275-L277)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/gcp/gcs_storage.go#L299-L301](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/gcp/gcs_storage.go#L299-L301)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L223-L225](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L223-L225)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L108-L110](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L108-L110)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/retry/retry.go#L193-L195](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/retry/retry.go#L193-L195)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/retry/retry.go#L171-L173](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/retry/retry.go#L171-L173)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/retry/retry.go#L194-L196](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/retry/retry.go#L194-L196)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L109-L111](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L109-L111)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L226-L228](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/cloud_io.go#L226-L228)
[https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/gcp/gcs_storage.go#L290-L292](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/cloud/gcp/gcs_storage.go#L290-L292)
[cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go#L117-L119](cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go#L117-L119)
[cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L886-L888](cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L886-L888)
[cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L832-L834](cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L832-L834)
[cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L64-L66](cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L64-L66)
[cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L32-L34](cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L32-L34)
[cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L39-L41](cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L39-L41)
[cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L66-L68](cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L66-L68)
[cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L833-L835](cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L833-L835)
[GOROOT/src/net/http/client.go#L580-L582](GOROOT/src/net/http/client.go#L580-L582)
[GOROOT/src/net/http/client.go#L714-L716](GOROOT/src/net/http/client.go#L714-L716)
[GOROOT/src/net/http/client.go#L174-L176](GOROOT/src/net/http/client.go#L174-L176)
[GOROOT/src/net/http/client.go#L250-L252](GOROOT/src/net/http/client.go#L250-L252)
[golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go#L44-L46](golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go#L44-L46)
[golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go#L303-L305](golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go#L303-L305)
[golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81](golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81)
[GOROOT/src/runtime/signal_unix.go#L838-L840](GOROOT/src/runtime/signal_unix.go#L838-L840)
[GOROOT/src/runtime/panic.go#L259-L261](GOROOT/src/runtime/panic.go#L259-L261)
[GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885)

```
pkg/ccl/backupccl/backup_job.go in pkg/ccl/backupccl.(*backupResumer).Resume at line 503
pkg/ccl/backupccl/backupdest/backup_destination.go in pkg/ccl/backupccl/backupdest.ResolveDest at line 169
pkg/ccl/backupccl/backupdest/backup_destination.go in pkg/ccl/backupccl/backupdest.containsManifest at line 71
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).ReadFile at line 331
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile at line 276
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFileAt at line 300
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open at line 224
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry at line 109
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts at line 194
pkg/util/retry/retry.go in pkg/util/retry.Options.Do at line 172
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts.func1 at line 195
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry.func1 at line 110
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open.func1 at line 227
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFileAt.func1 at line 291
cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go in cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader at line 118
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader at line 887
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2 at line 833
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run at line 65
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.Retry at line 33
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.retry at line 40
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run.func1 at line 67
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1 at line 834
GOROOT/src/net/http/client.go in net/http.(*Client).Do at line 581
GOROOT/src/net/http/client.go in net/http.(*Client).do at line 715
GOROOT/src/net/http/client.go in net/http.(*Client).send at line 175
GOROOT/src/net/http/client.go in net/http.send at line 251
golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go in golang.org/x/oauth2.(*Transport).RoundTrip at line 45
golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go in golang.org/x/oauth2.(*reuseTokenSource).Token at line 304
golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go in golang.org/x/oauth2/authhandler.authHandlerSource.Token at line 80
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 839
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
```

### Tags

| Tag | Value |
| --- | --- |
| Command | mt start-sql |
| Environment | v23.1.13 |
| Go Version | go1.19.13 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v23.1.13 |
| Cockroach SHA | 8d065df26e25a762a1abbbf2b1e475456b341a00 |
| # of CPUs | 16 |
| # of Goroutines | 256 |

Jira issue: CRDB-35073