cockroachdb / cockroachdb/cockroach
Sentry: replica_proposal.go:810: log.Fatal: while ingesting ×: pebble: backing file 000011 error: × (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.li...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7247973901/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7247973901/?referrer=webhooks_plugin)
Panic Message:
```
replica_proposal.go:810: log.Fatal: while ingesting ×: pebble: backing file 000011 error: ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.linkExternalSStablePreApply
| pkg/kv/kvserver/replica_proposal.go:810
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*appBatch).runPostAddTriggers
| pkg/kv/kvserver/app_batch.go:194
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| pkg/kv/kvserver/replica_app_batch.go:148
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| pkg/kv/kvserver/apply/cmd.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| pkg/kv/kvserver/apply/task.go:279
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| pkg/kv/kvserver/apply/task.go:246
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1234
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:869
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:761
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:411
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:302
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| pebble: backing file 000011 error: ×
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:468
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:191
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:84
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateTable
| | external/com_github_cockroachdb_pebble/file_cache.go:260
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:553
| | github.com/cockroachdb/pebble.(*tableCompaction).newRangeDelIter
| | external/com_github_cockroachdb_pebble/compaction.go:1249
| | github.com/cockroachdb/pebble.(*tableCompaction).newInputIters.func2
| | external/com_github_cockroachdb_pebble/compaction.go:1129
| | github.com/cockroachdb/pebble.(*tableCompaction).newInputIters
| | external/com_github_cockroachdb_pebble/compaction.go:1187
| | github.com/cockroachdb/pebble.(*DB).compactAndWrite
| | external/com_github_cockroachdb_pebble/compaction.go:3327
| | github.com/cockroachdb/pebble.(*DB).runCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:3237
| | github.com/cockroachdb/pebble.(*DB).compact1
| | external/com_github_cockroachdb_pebble/compaction.go:2691
| | github.com/cockroachdb/pebble.(*tableCompaction).Execute
| | external/com_github_cockroachdb_pebble/compaction.go:380
| | github.com/cockroachdb/pebble.(*DB).compact.func1.1
| | external/com_github_cockroachdb_pebble/compaction.go:2565
| | github.com/cockroachdb/pebble.(*DB).compact.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2589
| | runtime/pprof.Do
| | GOROOT/src/runtime/pprof/runtime.go:51
| | github.com/cockroachdb/pebble.(*DB).compact
| | external/com_github_cockroachdb_pebble/compaction.go:2559
| | github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2088
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (2) pebble: backing file 000011 error
| Wraps: (3) ×
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString
Wraps: (3) log.Fatal: while ingesting ×: pebble: backing file 000011 error: ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: while ingesting ×: pebble: backing file 000011 error: ×
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
replica_proposal.go:810: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/kv/kvserver/scheduler.go#L301-L303](pkg/kv/kvserver/scheduler.go#L301-L303)
[pkg/kv/kvserver/scheduler.go#L410-L412](pkg/kv/kvserver/scheduler.go#L410-L412)
[pkg/kv/kvserver/store_raft.go#L760-L762](pkg/kv/kvserver/store_raft.go#L760-L762)
[pkg/kv/kvserver/replica_raft.go#L868-L870](pkg/kv/kvserver/replica_raft.go#L868-L870)
[pkg/kv/kvserver/replica_raft.go#L1233-L1235](pkg/kv/kvserver/replica_raft.go#L1233-L1235)
[pkg/kv/kvserver/apply/task.go#L245-L247](pkg/kv/kvserver/apply/task.go#L245-L247)
[pkg/kv/kvserver/apply/task.go#L278-L280](pkg/kv/kvserver/apply/task.go#L278-L280)
[pkg/kv/kvserver/apply/cmd.go#L186-L188](pkg/kv/kvserver/apply/cmd.go#L186-L188)
[pkg/kv/kvserver/replica_app_batch.go#L147-L149](pkg/kv/kvserver/replica_app_batch.go#L147-L149)
[pkg/kv/kvserver/app_batch.go#L193-L195](pkg/kv/kvserver/app_batch.go#L193-L195)
[pkg/kv/kvserver/replica_proposal.go#L809-L811](pkg/kv/kvserver/replica_proposal.go#L809-L811)
```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 411
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 761
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 869
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1234
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 246
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 279
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCmdIter at line 187
pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).Stage at line 148
pkg/kv/kvserver/app_batch.go in pkg/kv/kvserver.(*appBatch).runPostAddTriggers at line 194
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.linkExternalSStablePreApply at line 810
```
### Tags
| Tag | Value |
| --- | --- |
| Command | demo |
| Environment | v25.3.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | darwin arm64 |
| Distribution | CCL |
| Cockroach Release | v25.3.2 |
| Cockroach SHA | a5e060ee11ad11e1649e7569486b579e9da8c8d7 |
| # of CPUs | 12 |
| # of Goroutines | 552 |
Jira issue: CRDB-59557
Contributor guide
Assessment
This issue has not been assessed yet.