cockroachdb / cockroachdb/cockroach

Sentry: panic.go:791: panic: × (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop ...

Open
#175,312 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7728929518/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7728929518/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:791: panic: ×
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
| pkg/util/stop/stopper.go:537
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| github.com/cockroachdb/cockroach/pkg/storage.newIntentInterleavingIterator
| pkg/storage/intent_interleaving_iter.go:250
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingReader).NewMVCCIterator
| pkg/storage/intent_interleaving_iter.go:75
| github.com/cockroachdb/cockroach/pkg/storage.iterateOnReader
| pkg/storage/engine.go:1814
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).MVCCIterate
| pkg/storage/pebble.go:1261
| github.com/cockroachdb/cockroach/pkg/cli.runSyncer
| debug_synctest.go:148
| github.com/cockroachdb/cockroach/pkg/cli.runDebugSyncTest
| debug_synctest.go:92
| github.com/spf13/cobra.(*Command).execute
| external/com_github_spf13_cobra/command.go:916
| github.com/spf13/cobra.(*Command).ExecuteC
| external/com_github_spf13_cobra/command.go:1044
| github.com/spf13/cobra.(*Command).Execute
| external/com_github_spf13_cobra/command.go:968
| github.com/cockroachdb/cockroach/pkg/cli.Run
| cli.go:301
| github.com/cockroachdb/cockroach/pkg/cli.doMain
| cli.go:139
| github.com/cockroachdb/cockroach/pkg/cli.Main
| cli.go:64
| main.main
| pkg/cmd/cockroach/main.go:21
| runtime.main
| GOROOT/src/runtime/proc.go:272
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) panic: ×
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: panic: ×
panic.go:791: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[GOROOT/src/runtime/proc.go#L271-L273](GOROOT/src/runtime/proc.go#L271-L273)
[pkg/cmd/cockroach/main.go#L20-L22](pkg/cmd/cockroach/main.go#L20-L22)
[cli.go#L63-L65](cli.go#L63-L65)
[cli.go#L138-L140](cli.go#L138-L140)
[cli.go#L300-L302](cli.go#L300-L302)
[external/com_github_spf13_cobra/command.go#L967-L969](external/com_github_spf13_cobra/command.go#L967-L969)
[external/com_github_spf13_cobra/command.go#L1043-L1045](external/com_github_spf13_cobra/command.go#L1043-L1045)
[external/com_github_spf13_cobra/command.go#L915-L917](external/com_github_spf13_cobra/command.go#L915-L917)
[debug_synctest.go#L91-L93](debug_synctest.go#L91-L93)
[debug_synctest.go#L147-L149](debug_synctest.go#L147-L149)
[pkg/storage/pebble.go#L1260-L1262](pkg/storage/pebble.go#L1260-L1262)
[pkg/storage/engine.go#L1813-L1815](pkg/storage/engine.go#L1813-L1815)
[pkg/storage/intent_interleaving_iter.go#L74-L76](pkg/storage/intent_interleaving_iter.go#L74-L76)
[pkg/storage/intent_interleaving_iter.go#L249-L251](pkg/storage/intent_interleaving_iter.go#L249-L251)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[pkg/util/stop/stopper.go#L536-L538](pkg/util/stop/stopper.go#L536-L538)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
GOROOT/src/runtime/proc.go in runtime.main at line 272
pkg/cmd/cockroach/main.go in main.main at line 21
cli.go in pkg/cli.Main at line 64
cli.go in pkg/cli.doMain at line 139
cli.go in pkg/cli.Run at line 301
external/com_github_spf13_cobra/command.go in github.com/spf13/cobra.(*Command).Execute at line 968
external/com_github_spf13_cobra/command.go in github.com/spf13/cobra.(*Command).ExecuteC at line 1044
external/com_github_spf13_cobra/command.go in github.com/spf13/cobra.(*Command).execute at line 916
debug_synctest.go in pkg/cli.runDebugSyncTest at line 92
debug_synctest.go in pkg/cli.runSyncer at line 148
pkg/storage/pebble.go in pkg/storage.(*Pebble).MVCCIterate at line 1261
pkg/storage/engine.go in pkg/storage.iterateOnReader at line 1814
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingReader).NewMVCCIterator at line 75
pkg/storage/intent_interleaving_iter.go in pkg/storage.newIntentInterleavingIterator at line 250
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).Stop at line 537
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
```

### Tags

| Tag | Value |
| --- | --- |
| Command | debug synctest |
| Environment | v25.2.0 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.0 |
| Cockroach SHA | 22cec2232cf3ab2a0e5f93cd35aa1217d62a4400 |
| # of CPUs | 16 |
| # of Goroutines | 31 |

Jira issue: CRDB-68339

Contributor guide

Open the contributing guide

Research direction

Start with the debug synctest stack in pkg/cli/debug_synctest.go, then trace the iterator path through pkg/storage/pebble.go, pkg/storage/engine.go, and pkg/storage/intent_interleaving_iter.go, including Stopper.Stop in pkg/util/stop/stopper.go. Reproduce the panic if possible and determine the failure condition; done means the debug synctest command no longer panics and a regression test covers the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.