cockroachdb / cockroachdb/cockroach
Sentry: panic.go:791: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | runtime.p...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/6777981727/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6777981727/?referrer=webhooks_plugin)
Panic Message:
```
panic.go:791: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| runtime.panicmem
| GOROOT/src/runtime/panic.go:262
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:917
| golang.org/x/oauth2/authhandler.authHandlerSource.Token
| external/org_golang_x_oauth2/authhandler/authhandler.go:80
| golang.org/x/oauth2.(*reuseTokenSource).Token
| external/org_golang_x_oauth2/oauth2.go:316
| golang.org/x/oauth2.(*Transport).RoundTrip
| external/org_golang_x_oauth2/transport.go:45
| net/http.send
| GOROOT/src/net/http/client.go:259
| net/http.(*Client).send
| GOROOT/src/net/http/client.go:180
| net/http.(*Client).do
| GOROOT/src/net/http/client.go:731
| net/http.(*Client).Do
| GOROOT/src/net/http/client.go:590
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1
| external/com_google_cloud_go_storage/http_client.go:834
| cloud.google.com/go/storage.run.func1
| external/com_google_cloud_go_storage/invoke.go:67
| cloud.google.com/go/internal.retry
| external/com_google_cloud_go/internal/retry.go:40
| cloud.google.com/go/internal.Retry
| external/com_google_cloud_go/internal/retry.go:33
| cloud.google.com/go/storage.run
| external/com_google_cloud_go_storage/invoke.go:65
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2
| external/com_google_cloud_go_storage/http_client.go:833
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader
| external/com_google_cloud_go_storage/http_client.go:887
| cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader
| external/com_google_cloud_go_storage/reader.go:118
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile.func1
| pkg/cloud/gcp/gcs_storage.go:310
| github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open.func1
| pkg/cloud/cloud_io.go:273
| github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry.func1
| pkg/cloud/cloud_io.go:143
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
| pkg/util/retry/retry.go:195
| github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
| pkg/util/retry/retry.go:172
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
| pkg/util/retry/retry.go:194
| github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry
| pkg/cloud/cloud_io.go:142
| github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open
| pkg/cloud/cloud_io.go:271
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile
| pkg/cloud/gcp/gcs_storage.go:324
| github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).ReadFile
| pkg/cloud/impl_registry.go:399
| github.com/cockroachdb/cockroach/pkg/backup/backupdest.containsManifest
| pkg/backup/backupdest/backup_destination.go:70
| github.com/cockroachdb/cockroach/pkg/backup/backupdest.ResolveDest
| pkg/backup/backupdest/backup_destination.go:162
| github.com/cockroachdb/cockroach/pkg/backup.(*backupResumer).Resume
| pkg/backup/backup_job.go:579
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| pkg/jobs/registry.go:1720
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:791: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[pkg/jobs/registry.go#L1719-L1721](pkg/jobs/registry.go#L1719-L1721)
[pkg/backup/backup_job.go#L578-L580](pkg/backup/backup_job.go#L578-L580)
[pkg/backup/backupdest/backup_destination.go#L161-L163](pkg/backup/backupdest/backup_destination.go#L161-L163)
[pkg/backup/backupdest/backup_destination.go#L69-L71](pkg/backup/backupdest/backup_destination.go#L69-L71)
[pkg/cloud/impl_registry.go#L398-L400](pkg/cloud/impl_registry.go#L398-L400)
[pkg/cloud/gcp/gcs_storage.go#L323-L325](pkg/cloud/gcp/gcs_storage.go#L323-L325)
[pkg/cloud/cloud_io.go#L270-L272](pkg/cloud/cloud_io.go#L270-L272)
[pkg/cloud/cloud_io.go#L141-L143](pkg/cloud/cloud_io.go#L141-L143)
[pkg/util/retry/retry.go#L193-L195](pkg/util/retry/retry.go#L193-L195)
[pkg/util/retry/retry.go#L171-L173](pkg/util/retry/retry.go#L171-L173)
[pkg/util/retry/retry.go#L194-L196](pkg/util/retry/retry.go#L194-L196)
[pkg/cloud/cloud_io.go#L142-L144](pkg/cloud/cloud_io.go#L142-L144)
[pkg/cloud/cloud_io.go#L272-L274](pkg/cloud/cloud_io.go#L272-L274)
[pkg/cloud/gcp/gcs_storage.go#L309-L311](pkg/cloud/gcp/gcs_storage.go#L309-L311)
[external/com_google_cloud_go_storage/reader.go#L117-L119](external/com_google_cloud_go_storage/reader.go#L117-L119)
[external/com_google_cloud_go_storage/http_client.go#L886-L888](external/com_google_cloud_go_storage/http_client.go#L886-L888)
[external/com_google_cloud_go_storage/http_client.go#L832-L834](external/com_google_cloud_go_storage/http_client.go#L832-L834)
[external/com_google_cloud_go_storage/invoke.go#L64-L66](external/com_google_cloud_go_storage/invoke.go#L64-L66)
[external/com_google_cloud_go/internal/retry.go#L32-L34](external/com_google_cloud_go/internal/retry.go#L32-L34)
[external/com_google_cloud_go/internal/retry.go#L39-L41](external/com_google_cloud_go/internal/retry.go#L39-L41)
[external/com_google_cloud_go_storage/invoke.go#L66-L68](external/com_google_cloud_go_storage/invoke.go#L66-L68)
[external/com_google_cloud_go_storage/http_client.go#L833-L835](external/com_google_cloud_go_storage/http_client.go#L833-L835)
[GOROOT/src/net/http/client.go#L589-L591](GOROOT/src/net/http/client.go#L589-L591)
[GOROOT/src/net/http/client.go#L730-L732](GOROOT/src/net/http/client.go#L730-L732)
[GOROOT/src/net/http/client.go#L179-L181](GOROOT/src/net/http/client.go#L179-L181)
[GOROOT/src/net/http/client.go#L258-L260](GOROOT/src/net/http/client.go#L258-L260)
[external/org_golang_x_oauth2/transport.go#L44-L46](external/org_golang_x_oauth2/transport.go#L44-L46)
[external/org_golang_x_oauth2/oauth2.go#L315-L317](external/org_golang_x_oauth2/oauth2.go#L315-L317)
[external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81](external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81)
[GOROOT/src/runtime/signal_unix.go#L916-L918](GOROOT/src/runtime/signal_unix.go#L916-L918)
[GOROOT/src/runtime/panic.go#L261-L263](GOROOT/src/runtime/panic.go#L261-L263)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
```
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1720
pkg/backup/backup_job.go in pkg/backup.(*backupResumer).Resume at line 579
pkg/backup/backupdest/backup_destination.go in pkg/backup/backupdest.ResolveDest at line 162
pkg/backup/backupdest/backup_destination.go in pkg/backup/backupdest.containsManifest at line 70
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).ReadFile at line 399
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile at line 324
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open at line 271
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry at line 142
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts at line 194
pkg/util/retry/retry.go in pkg/util/retry.Options.Do at line 172
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts.func1 at line 195
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry.func1 at line 143
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open.func1 at line 273
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile.func1 at line 310
external/com_google_cloud_go_storage/reader.go in cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader at line 118
external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader at line 887
external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2 at line 833
external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run at line 65
external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.Retry at line 33
external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.retry at line 40
external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run.func1 at line 67
external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1 at line 834
GOROOT/src/net/http/client.go in net/http.(*Client).Do at line 590
GOROOT/src/net/http/client.go in net/http.(*Client).do at line 731
GOROOT/src/net/http/client.go in net/http.(*Client).send at line 180
GOROOT/src/net/http/client.go in net/http.send at line 259
external/org_golang_x_oauth2/transport.go in golang.org/x/oauth2.(*Transport).RoundTrip at line 45
external/org_golang_x_oauth2/oauth2.go in golang.org/x/oauth2.(*reuseTokenSource).Token at line 316
external/org_golang_x_oauth2/authhandler/authhandler.go in golang.org/x/oauth2/authhandler.authHandlerSource.Token at line 80
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 917
GOROOT/src/runtime/panic.go in runtime.panicmem at line 262
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
```
### Tags
| Tag | Value |
| --- | --- |
| Command | mt start-sql |
| Environment | v25.2.1 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.1 |
| Cockroach SHA | f9f183de9d58b822d6cf4c12c560e1fbbc70707f |
| # of CPUs | 16 |
| # of Goroutines | 2427 |
Jira issue: CRDB-53021
Contributor guide
Research direction
Start by reading pkg/cloud/gcp/gcs_storage.go around ReadFile and pkg/cloud/cloud_io.go around ResumingReader.Open, then trace the call from pkg/backup/backupdest/backup_destination.go. Reproduce or inspect the GCS-backed backup path for the OAuth token panic, and add coverage or a fix that prevents this crash while preserving backup destination resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100