cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1373: log.Dev.Fatal: local corruption detected: pebble: file 019163: block 0/8348: × checksum mismatch 6d096709 != 28c6a7be (1) attached stack trace -- stack trace: | github.com...

Open
#170,936 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.1 C-bug O-sentry
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7507405211/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7507405211/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1373: log.Dev.Fatal: local corruption detected: pebble: file 019163: block 0/8348: × checksum mismatch 6d096709 != 28c6a7be
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1373
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:323
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:418
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:391
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:508
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).firstInternal
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1344
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).First
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1286
| github.com/cockroachdb/pebble.(*levelIter).internalFirst
| external/com_github_cockroachdb_pebble/level_iter.go:790
| github.com/cockroachdb/pebble.(*levelIter).First
| external/com_github_cockroachdb_pebble/level_iter.go:758
| github.com/cockroachdb/pebble.(*mergingIter).First
| external/com_github_cockroachdb_pebble/merging_iter.go:1185
| github.com/cockroachdb/pebble/internal/compact.(*Iter).First
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:423
| github.com/cockroachdb/pebble/internal/compact.NewRunner
| external/com_github_cockroachdb_pebble/internal/compact/run.go:149
| github.com/cockroachdb/pebble.(*DB).compactAndWrite
| external/com_github_cockroachdb_pebble/compaction.go:2848
| github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| external/com_github_cockroachdb_pebble/compaction.go:2713
| github.com/cockroachdb/pebble.(*DB).runCompaction
| external/com_github_cockroachdb_pebble/compaction.go:2695
| github.com/cockroachdb/pebble.(*DB).compact1
| external/com_github_cockroachdb_pebble/compaction.go:2378
| github.com/cockroachdb/pebble.(*tableCompaction).Execute
| external/com_github_cockroachdb_pebble/compaction.go:410
| github.com/cockroachdb/pebble.(*DB).compact.func1.1
| external/com_github_cockroachdb_pebble/compaction.go:2246
| github.com/cockroachdb/pebble.(*DB).compact.func1
| external/com_github_cockroachdb_pebble/compaction.go:2275
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble.(*DB).compact
| external/com_github_cockroachdb_pebble/compaction.go:2240
| github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| external/com_github_cockroachdb_pebble/compaction.go:2109
| runtime.goexit
| src/runtime/asm_arm64.s:1268
Wraps: (2) secondary error attachment
| pebble: file 019163: block 0/8348: × checksum mismatch 6d096709 != 28c6a7be
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:532
| | [...repeated from below...]
| Wraps: (3) pebble: file 019163
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:193
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:530
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:416
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:391
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:508
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).firstInternal
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1344
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).First
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1286
| | github.com/cockroachdb/pebble.(*levelIter).internalFirst
| | external/com_github_cockroachdb_pebble/level_iter.go:790
| | github.com/cockroachdb/pebble.(*levelIter).First
| | external/com_github_cockroachdb_pebble/level_iter.go:758
| | github.com/cockroachdb/pebble.(*mergingIter).First
| | external/com_github_cockroachdb_pebble/merging_iter.go:1185
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).First
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:423
| | github.com/cockroachdb/pebble/internal/compact.NewRunner
| | external/com_github_cockroachdb_pebble/internal/compact/run.go:149
| | github.com/cockroachdb/pebble.(*DB).compactAndWrite
| | external/com_github_cockroachdb_pebble/compaction.go:2848
| | github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:2713
| | github.com/cockroachdb/pebble.(*DB).runCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:2695
| | github.com/cockroachdb/pebble.(*DB).compact1
| | external/com_github_cockroachdb_pebble/compaction.go:2378
| | github.com/cockroachdb/pebble.(*tableCompaction).Execute
| | external/com_github_cockroachdb_pebble/compaction.go:410
| | github.com/cockroachdb/pebble.(*DB).compact.func1.1
| | external/com_github_cockroachdb_pebble/compaction.go:2246
| | github.com/cockroachdb/pebble.(*DB).compact.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2275
| | runtime/pprof.Do
| | GOROOT/src/runtime/pprof/runtime.go:51
| | github.com/cockroachdb/pebble.(*DB).compact
| | external/com_github_cockroachdb_pebble/compaction.go:2240
| | github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2109
| | runtime.goexit
| | src/runtime/asm_arm64.s:1268
| Wraps: (6) block 0/8348: × checksum mismatch 6d096709 != 28c6a7be
| Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Dev.Fatal: local corruption detected: pebble: file 019163: block 0/8348: × checksum mismatch 6d096709 != 28c6a7be
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:532
| | [...repeated from below...]
| Wraps: (3) pebble: file 019163
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:193
| | github.com/cockroachdb/pebble/sstable/block.(*Reader)....
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1267-L1269](src/runtime/asm_arm64.s#L1267-L1269)
[external/com_github_cockroachdb_pebble/compaction.go#L2108-L2110](external/com_github_cockroachdb_pebble/compaction.go#L2108-L2110)
[external/com_github_cockroachdb_pebble/compaction.go#L2239-L2241](external/com_github_cockroachdb_pebble/compaction.go#L2239-L2241)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/compaction.go#L2274-L2276](external/com_github_cockroachdb_pebble/compaction.go#L2274-L2276)
[external/com_github_cockroachdb_pebble/compaction.go#L2245-L2247](external/com_github_cockroachdb_pebble/compaction.go#L2245-L2247)
[external/com_github_cockroachdb_pebble/compaction.go#L409-L411](external/com_github_cockroachdb_pebble/compaction.go#L409-L411)
[external/com_github_cockroachdb_pebble/compaction.go#L2377-L2379](external/com_github_cockroachdb_pebble/compaction.go#L2377-L2379)
[external/com_github_cockroachdb_pebble/compaction.go#L2694-L2696](external/com_github_cockroachdb_pebble/compaction.go#L2694-L2696)
[external/com_github_cockroachdb_pebble/compaction.go#L2712-L2714](external/com_github_cockroachdb_pebble/compaction.go#L2712-L2714)
[external/com_github_cockroachdb_pebble/compaction.go#L2847-L2849](external/com_github_cockroachdb_pebble/compaction.go#L2847-L2849)
[external/com_github_cockroachdb_pebble/internal/compact/run.go#L148-L150](external/com_github_cockroachdb_pebble/internal/compact/run.go#L148-L150)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L422-L424](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L422-L424)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1184-L1186](external/com_github_cockroachdb_pebble/merging_iter.go#L1184-L1186)
[external/com_github_cockroachdb_pebble/level_iter.go#L757-L759](external/com_github_cockroachdb_pebble/level_iter.go#L757-L759)
[external/com_github_cockroachdb_pebble/level_iter.go#L789-L791](external/com_github_cockroachdb_pebble/level_iter.go#L789-L791)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1285-L1287](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1285-L1287)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1343-L1345](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1343-L1345)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L507-L509](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L507-L509)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L390-L392](external/com_github_cockroachdb_pebble/sstable/reader.go#L390-L392)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L417-L419](external/com_github_cockroachdb_pebble/sstable/block/block.go#L417-L419)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L322-L324](external/com_github_cockroachdb_pebble/sstable/block/block.go#L322-L324)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1372-L1374](pkg/storage/pebble.go#L1372-L1374)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1268
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1 at line 2109
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact at line 2240
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1 at line 2275
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1.1 at line 2246
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*tableCompaction).Execute at line 410
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact1 at line 2378
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runCompaction at line 2695
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction at line 2713
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compactAndWrite at line 2848
external/com_github_cockroachdb_pebble/internal/compact/run.go in github.com/cockroachdb/pebble/internal/compact.NewRunner at line 149
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).First at line 423
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).First at line 1185
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).First at line 758
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).internalFirst at line 790
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).First at line 1286
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).firstInternal at line 1344
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 508
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 391
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 418
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 323
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1373
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v26.1.4 |
| Go Version | go1.25.5 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.1.4 |
| Cockroach SHA | 87953f064653dfa86d766ecfc6f475e78a59fead |
| # of CPUs | 14 |
| # of Goroutines | 549 |

Jira issue: CRDB-64248

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.