cockroachdb / cockroachdb/cockroach
Sentry: pebble.go:1436: log.Dev.Fatal: local corruption detected: pebble: file 000016: block 237440/12: × checksum mismatch fd0df90d != a1ac617 (1) attached stack trace -- stack trace: | github.c...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7569470984/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7569470984/?referrer=webhooks_plugin)
Panic Message:
```
pebble.go:1436: log.Dev.Fatal: local corruption detected: pebble: file 000016: block 237440/12: × checksum mismatch fd0df90d != a1ac617
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1436
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:331
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:426
| github.com/cockroachdb/pebble/sstable.(*Reader).readValueBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:456
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).ReadValueBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:530
| github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).getValueInternal
| external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:275
| github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).FetchHandle
| external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:218
| github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| external/com_github_cockroachdb_pebble/internal/base/value.go:79
| github.com/cockroachdb/pebble/internal/base.(*InternalKV).Value
| external/com_github_cockroachdb_pebble/internal/base/internal.go:718
| github.com/cockroachdb/pebble/internal/compact.(*Iter).saveValue
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:1400
| github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:873
| github.com/cockroachdb/pebble/internal/compact.(*Iter).Next
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:633
| github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable
| external/com_github_cockroachdb_pebble/internal/compact/run.go:264
| github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable
| external/com_github_cockroachdb_pebble/internal/compact/run.go:201
| github.com/cockroachdb/pebble.(*DB).compactAndWrite
| external/com_github_cockroachdb_pebble/compaction.go:2902
| github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| external/com_github_cockroachdb_pebble/compaction.go:2723
| github.com/cockroachdb/pebble.(*DB).runCompaction
| external/com_github_cockroachdb_pebble/compaction.go:2705
| github.com/cockroachdb/pebble.(*DB).compact1
| external/com_github_cockroachdb_pebble/compaction.go:2387
| github.com/cockroachdb/pebble.(*tableCompaction).Execute
| external/com_github_cockroachdb_pebble/compaction.go:408
| github.com/cockroachdb/pebble.(*DB).compact.func1.1
| external/com_github_cockroachdb_pebble/compaction.go:2255
| github.com/cockroachdb/pebble.(*DB).compact.func1
| external/com_github_cockroachdb_pebble/compaction.go:2284
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble.(*DB).compact
| external/com_github_cockroachdb_pebble/compaction.go:2249
| github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| external/com_github_cockroachdb_pebble/compaction.go:2118
| runtime.goexit
| src/runtime/asm_arm64.s:1268
Wraps: (2) secondary error attachment
| pebble: file 000016: block 237440/12: × checksum mismatch fd0df90d != a1ac617
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:540
| | [...repeated from below...]
| Wraps: (3) pebble: file 000016
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:193
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:538
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:424
| | github.com/cockroachdb/pebble/sstable.(*Reader).readValueBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:456
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).ReadValueBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:530
| | github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).getValueInternal
| | external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:275
| | github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).FetchHandle
| | external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:218
| | github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| | github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/value.go:79
| | github.com/cockroachdb/pebble/internal/base.(*InternalKV).Value
| | external/com_github_cockroachdb_pebble/internal/base/internal.go:718
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).saveValue
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:1400
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:873
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).Next
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:633
| | github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable
| | external/com_github_cockroachdb_pebble/internal/compact/run.go:264
| | github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable
| | external/com_github_cockroachdb_pebble/internal/compact/run.go:201
| | github.com/cockroachdb/pebble.(*DB).compactAndWrite
| | external/com_github_cockroachdb_pebble/compaction.go:2902
| | github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:2723
| | github.com/cockroachdb/pebble.(*DB).runCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:2705
| | github.com/cockroachdb/pebble.(*DB).compact1
| | external/com_github_cockroachdb_pebble/compaction.go:2387
| | github.com/cockroachdb/pebble.(*tableCompaction).Execute
| | external/com_github_cockroachdb_pebble/compaction.go:408
| | github.com/cockroachdb/pebble.(*DB).compact.func1.1
| | external/com_github_cockroachdb_pebble/compaction.go:2255
| | github.com/cockroachdb/pebble.(*DB).compact.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2284
| | runtime/pprof.Do
| | GOROOT/src/runtime/pprof/runtime.go:51
| | github.com/cockroachdb/pebble.(*DB).compact
| | external/com_github_cockroachdb_pebble/compaction.go:2249
| | github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2118
| | runtime.goexit
| | src/runtime/asm_arm64.s:1268
| Wraps: (6) block 237440/12: × checksum mismatch fd0df90d != a1ac617
| Error types: (1) *hintdetail.withHint (2...
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_arm64.s#L1267-L1269](src/runtime/asm_arm64.s#L1267-L1269)
[external/com_github_cockroachdb_pebble/compaction.go#L2117-L2119](external/com_github_cockroachdb_pebble/compaction.go#L2117-L2119)
[external/com_github_cockroachdb_pebble/compaction.go#L2248-L2250](external/com_github_cockroachdb_pebble/compaction.go#L2248-L2250)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/compaction.go#L2283-L2285](external/com_github_cockroachdb_pebble/compaction.go#L2283-L2285)
[external/com_github_cockroachdb_pebble/compaction.go#L2254-L2256](external/com_github_cockroachdb_pebble/compaction.go#L2254-L2256)
[external/com_github_cockroachdb_pebble/compaction.go#L407-L409](external/com_github_cockroachdb_pebble/compaction.go#L407-L409)
[external/com_github_cockroachdb_pebble/compaction.go#L2386-L2388](external/com_github_cockroachdb_pebble/compaction.go#L2386-L2388)
[external/com_github_cockroachdb_pebble/compaction.go#L2704-L2706](external/com_github_cockroachdb_pebble/compaction.go#L2704-L2706)
[external/com_github_cockroachdb_pebble/compaction.go#L2722-L2724](external/com_github_cockroachdb_pebble/compaction.go#L2722-L2724)
[external/com_github_cockroachdb_pebble/compaction.go#L2901-L2903](external/com_github_cockroachdb_pebble/compaction.go#L2901-L2903)
[external/com_github_cockroachdb_pebble/internal/compact/run.go#L200-L202](external/com_github_cockroachdb_pebble/internal/compact/run.go#L200-L202)
[external/com_github_cockroachdb_pebble/internal/compact/run.go#L263-L265](external/com_github_cockroachdb_pebble/internal/compact/run.go#L263-L265)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L632-L634](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L632-L634)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L872-L874](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L872-L874)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L1399-L1401](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L1399-L1401)
[external/com_github_cockroachdb_pebble/internal/base/internal.go#L717-L719](external/com_github_cockroachdb_pebble/internal/base/internal.go#L717-L719)
[external/com_github_cockroachdb_pebble/internal/base/value.go#L78-L80](external/com_github_cockroachdb_pebble/internal/base/value.go#L78-L80)
[external/com_github_cockroachdb_pebble/internal/base/lazy_value.go#L194-L196](external/com_github_cockroachdb_pebble/internal/base/lazy_value.go#L194-L196)
[external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L217-L219](external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L217-L219)
[external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L274-L276](external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L274-L276)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L529-L531](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L529-L531)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L455-L457](external/com_github_cockroachdb_pebble/sstable/reader.go#L455-L457)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L425-L427](external/com_github_cockroachdb_pebble/sstable/block/block.go#L425-L427)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L330-L332](external/com_github_cockroachdb_pebble/sstable/block/block.go#L330-L332)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1435-L1437](pkg/storage/pebble.go#L1435-L1437)
```
src/runtime/asm_arm64.s in runtime.goexit at line 1268
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1 at line 2118
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact at line 2249
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1 at line 2284
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1.1 at line 2255
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*tableCompaction).Execute at line 408
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact1 at line 2387
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runCompaction at line 2705
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction at line 2723
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compactAndWrite at line 2902
external/com_github_cockroachdb_pebble/internal/compact/run.go in github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable at line 201
external/com_github_cockroachdb_pebble/internal/compact/run.go in github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable at line 264
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).Next at line 633
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext at line 873
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).saveValue at line 1400
external/com_github_cockroachdb_pebble/internal/base/internal.go in github.com/cockroachdb/pebble/internal/base.(*InternalKV).Value at line 718
external/com_github_cockroachdb_pebble/internal/base/value.go in github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value at line 79
external/com_github_cockroachdb_pebble/internal/base/lazy_value.go in github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value at line 195
external/com_github_cockroachdb_pebble/sstable/valblk/reader.go in github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).FetchHandle at line 218
external/com_github_cockroachdb_pebble/sstable/valblk/reader.go in github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).getValueInternal at line 275
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).ReadValueBlock at line 530
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readValueBlock at line 456
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 426
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 331
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1436
```
### Tags
| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v26.2.0 |
| Go Version | go1.25.5 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.2.0 |
| Cockroach SHA | 3d2773444f3ed07326a5daf43bc1555457d6964b |
| # of CPUs | 8 |
| # of Goroutines | 454 |
Jira issue: CRDB-65030
Contributor guide
Research direction
Start with pkg/storage/pebble.go:1436 and the linked Sentry report, then trace the cited Pebble compaction and checksum-validation frames under external/com_github_cockroachdb_pebble. The report provides no reproduction steps or proposed change, so the cause and completion criteria must be established before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100