cockroachdb / cockroachdb/cockroach

Sentry: panic.go:791: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | runtime.p...

Open
#150,992 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery branch-release-25.2 C-bug O-sentry P-3 T-disaster-recovery X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6777981727/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6777981727/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:791: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| runtime.panicmem
| GOROOT/src/runtime/panic.go:262
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:917
| golang.org/x/oauth2/authhandler.authHandlerSource.Token
| external/org_golang_x_oauth2/authhandler/authhandler.go:80
| golang.org/x/oauth2.(*reuseTokenSource).Token
| external/org_golang_x_oauth2/oauth2.go:316
| golang.org/x/oauth2.(*Transport).RoundTrip
| external/org_golang_x_oauth2/transport.go:45
| net/http.send
| GOROOT/src/net/http/client.go:259
| net/http.(*Client).send
| GOROOT/src/net/http/client.go:180
| net/http.(*Client).do
| GOROOT/src/net/http/client.go:731
| net/http.(*Client).Do
| GOROOT/src/net/http/client.go:590
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1
| external/com_google_cloud_go_storage/http_client.go:834
| cloud.google.com/go/storage.run.func1
| external/com_google_cloud_go_storage/invoke.go:67
| cloud.google.com/go/internal.retry
| external/com_google_cloud_go/internal/retry.go:40
| cloud.google.com/go/internal.Retry
| external/com_google_cloud_go/internal/retry.go:33
| cloud.google.com/go/storage.run
| external/com_google_cloud_go_storage/invoke.go:65
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2
| external/com_google_cloud_go_storage/http_client.go:833
| cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader
| external/com_google_cloud_go_storage/http_client.go:887
| cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader
| external/com_google_cloud_go_storage/reader.go:118
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile.func1
| pkg/cloud/gcp/gcs_storage.go:310
| github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open.func1
| pkg/cloud/cloud_io.go:273
| github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry.func1
| pkg/cloud/cloud_io.go:143
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
| pkg/util/retry/retry.go:195
| github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
| pkg/util/retry/retry.go:172
| github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
| pkg/util/retry/retry.go:194
| github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry
| pkg/cloud/cloud_io.go:142
| github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open
| pkg/cloud/cloud_io.go:271
| github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile
| pkg/cloud/gcp/gcs_storage.go:324
| github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).ReadFile
| pkg/cloud/impl_registry.go:399
| github.com/cockroachdb/cockroach/pkg/backup/backupdest.containsManifest
| pkg/backup/backupdest/backup_destination.go:70
| github.com/cockroachdb/cockroach/pkg/backup/backupdest.ResolveDest
| pkg/backup/backupdest/backup_destination.go:162
| github.com/cockroachdb/cockroach/pkg/backup.(*backupResumer).Resume
| pkg/backup/backup_job.go:579
| github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
| pkg/jobs/registry.go:1720
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:791: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[pkg/jobs/registry.go#L1719-L1721](pkg/jobs/registry.go#L1719-L1721)
[pkg/backup/backup_job.go#L578-L580](pkg/backup/backup_job.go#L578-L580)
[pkg/backup/backupdest/backup_destination.go#L161-L163](pkg/backup/backupdest/backup_destination.go#L161-L163)
[pkg/backup/backupdest/backup_destination.go#L69-L71](pkg/backup/backupdest/backup_destination.go#L69-L71)
[pkg/cloud/impl_registry.go#L398-L400](pkg/cloud/impl_registry.go#L398-L400)
[pkg/cloud/gcp/gcs_storage.go#L323-L325](pkg/cloud/gcp/gcs_storage.go#L323-L325)
[pkg/cloud/cloud_io.go#L270-L272](pkg/cloud/cloud_io.go#L270-L272)
[pkg/cloud/cloud_io.go#L141-L143](pkg/cloud/cloud_io.go#L141-L143)
[pkg/util/retry/retry.go#L193-L195](pkg/util/retry/retry.go#L193-L195)
[pkg/util/retry/retry.go#L171-L173](pkg/util/retry/retry.go#L171-L173)
[pkg/util/retry/retry.go#L194-L196](pkg/util/retry/retry.go#L194-L196)
[pkg/cloud/cloud_io.go#L142-L144](pkg/cloud/cloud_io.go#L142-L144)
[pkg/cloud/cloud_io.go#L272-L274](pkg/cloud/cloud_io.go#L272-L274)
[pkg/cloud/gcp/gcs_storage.go#L309-L311](pkg/cloud/gcp/gcs_storage.go#L309-L311)
[external/com_google_cloud_go_storage/reader.go#L117-L119](external/com_google_cloud_go_storage/reader.go#L117-L119)
[external/com_google_cloud_go_storage/http_client.go#L886-L888](external/com_google_cloud_go_storage/http_client.go#L886-L888)
[external/com_google_cloud_go_storage/http_client.go#L832-L834](external/com_google_cloud_go_storage/http_client.go#L832-L834)
[external/com_google_cloud_go_storage/invoke.go#L64-L66](external/com_google_cloud_go_storage/invoke.go#L64-L66)
[external/com_google_cloud_go/internal/retry.go#L32-L34](external/com_google_cloud_go/internal/retry.go#L32-L34)
[external/com_google_cloud_go/internal/retry.go#L39-L41](external/com_google_cloud_go/internal/retry.go#L39-L41)
[external/com_google_cloud_go_storage/invoke.go#L66-L68](external/com_google_cloud_go_storage/invoke.go#L66-L68)
[external/com_google_cloud_go_storage/http_client.go#L833-L835](external/com_google_cloud_go_storage/http_client.go#L833-L835)
[GOROOT/src/net/http/client.go#L589-L591](GOROOT/src/net/http/client.go#L589-L591)
[GOROOT/src/net/http/client.go#L730-L732](GOROOT/src/net/http/client.go#L730-L732)
[GOROOT/src/net/http/client.go#L179-L181](GOROOT/src/net/http/client.go#L179-L181)
[GOROOT/src/net/http/client.go#L258-L260](GOROOT/src/net/http/client.go#L258-L260)
[external/org_golang_x_oauth2/transport.go#L44-L46](external/org_golang_x_oauth2/transport.go#L44-L46)
[external/org_golang_x_oauth2/oauth2.go#L315-L317](external/org_golang_x_oauth2/oauth2.go#L315-L317)
[external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81](external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81)
[GOROOT/src/runtime/signal_unix.go#L916-L918](GOROOT/src/runtime/signal_unix.go#L916-L918)
[GOROOT/src/runtime/panic.go#L261-L263](GOROOT/src/runtime/panic.go#L261-L263)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)

```
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1720
pkg/backup/backup_job.go in pkg/backup.(*backupResumer).Resume at line 579
pkg/backup/backupdest/backup_destination.go in pkg/backup/backupdest.ResolveDest at line 162
pkg/backup/backupdest/backup_destination.go in pkg/backup/backupdest.containsManifest at line 70
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).ReadFile at line 399
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile at line 324
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open at line 271
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry at line 142
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts at line 194
pkg/util/retry/retry.go in pkg/util/retry.Options.Do at line 172
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts.func1 at line 195
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry.func1 at line 143
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open.func1 at line 273
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile.func1 at line 310
external/com_google_cloud_go_storage/reader.go in cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader at line 118
external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader at line 887
external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2 at line 833
external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run at line 65
external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.Retry at line 33
external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.retry at line 40
external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run.func1 at line 67
external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1 at line 834
GOROOT/src/net/http/client.go in net/http.(*Client).Do at line 590
GOROOT/src/net/http/client.go in net/http.(*Client).do at line 731
GOROOT/src/net/http/client.go in net/http.(*Client).send at line 180
GOROOT/src/net/http/client.go in net/http.send at line 259
external/org_golang_x_oauth2/transport.go in golang.org/x/oauth2.(*Transport).RoundTrip at line 45
external/org_golang_x_oauth2/oauth2.go in golang.org/x/oauth2.(*reuseTokenSource).Token at line 316
external/org_golang_x_oauth2/authhandler/authhandler.go in golang.org/x/oauth2/authhandler.authHandlerSource.Token at line 80
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 917
GOROOT/src/runtime/panic.go in runtime.panicmem at line 262
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
```

### Tags

| Tag | Value |
| --- | --- |
| Command | mt start-sql |
| Environment | v25.2.1 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.1 |
| Cockroach SHA | f9f183de9d58b822d6cf4c12c560e1fbbc70707f |
| # of CPUs | 16 |
| # of Goroutines | 2427 |

Jira issue: CRDB-53021

Contributor guide

Open the contributing guide

Research direction

Start by reading pkg/cloud/gcp/gcs_storage.go around ReadFile and pkg/cloud/cloud_io.go around ResumingReader.Open, then trace the call from pkg/backup/backupdest/backup_destination.go. Reproduce or inspect the GCS-backed backup path for the OAuth token panic, and add coverage or a fix that prevents this crash while preserving backup destination resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.