cockroachdb / cockroachdb/cockroach
Sentry: pebble.go:1231: log.Fatal: local corruption detected: pebble: file 198033: block 3468620/587: × checksum mismatch 49e1ed5c != d1b3a80d (1) attached stack trace -- stack trace: | github.co...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7463948038/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7463948038/?referrer=webhooks_plugin)
Panic Message:
```
pebble.go:1231: log.Fatal: local corruption detected: pebble: file 198033: block 3468620/587: × checksum mismatch 49e1ed5c != d1b3a80d
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1231
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1174
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1374
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1333
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:435
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:534
| github.com/cockroachdb/pebble/sstable.(*Reader).readRangeDelBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:393
| github.com/cockroachdb/pebble/sstable.(*Reader).NewRawRangeDelIter
| external/com_github_cockroachdb_pebble/sstable/reader.go:253
| github.com/cockroachdb/pebble.newCombinedDeletionKeyspanIter
| external/com_github_cockroachdb_pebble/table_stats.go:982
| github.com/cockroachdb/pebble.(*DB).loadTableRangeDelStats
| external/com_github_cockroachdb_pebble/table_stats.go:384
| github.com/cockroachdb/pebble.(*DB).loadTableStats.func1
| external/com_github_cockroachdb_pebble/table_stats.go:335
| github.com/cockroachdb/pebble.(*fileCacheHandle).withReader
| external/com_github_cockroachdb_pebble/file_cache.go:381
| github.com/cockroachdb/pebble.(*DB).loadTableStats
| external/com_github_cockroachdb_pebble/table_stats.go:308
| github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats-range1
| external/com_github_cockroachdb_pebble/table_stats.go:285
| github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats.(*LevelMetadata).All.func1
| external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go:150
| github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats
| external/com_github_cockroachdb_pebble/table_stats.go:227
| github.com/cockroachdb/pebble.(*DB).collectTableStats
| external/com_github_cockroachdb_pebble/table_stats.go:119
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) secondary error attachment
| pebble: file 198033: block 3468620/587: × checksum mismatch 49e1ed5c != d1b3a80d
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:598
| | [...repeated from below...]
| Wraps: (3) pebble: file 198033
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:206
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:596
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:531
| | github.com/cockroachdb/pebble/sstable.(*Reader).readRangeDelBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:393
| | github.com/cockroachdb/pebble/sstable.(*Reader).NewRawRangeDelIter
| | external/com_github_cockroachdb_pebble/sstable/reader.go:253
| | github.com/cockroachdb/pebble.newCombinedDeletionKeyspanIter
| | external/com_github_cockroachdb_pebble/table_stats.go:982
| | github.com/cockroachdb/pebble.(*DB).loadTableRangeDelStats
| | external/com_github_cockroachdb_pebble/table_stats.go:384
| | github.com/cockroachdb/pebble.(*DB).loadTableStats.func1
| | external/com_github_cockroachdb_pebble/table_stats.go:335
| | github.com/cockroachdb/pebble.(*fileCacheHandle).withReader
| | external/com_github_cockroachdb_pebble/file_cache.go:381
| | github.com/cockroachdb/pebble.(*DB).loadTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:308
| | github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats-range1
| | external/com_github_cockroachdb_pebble/table_stats.go:285
| | github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats.(*LevelMetadata).All.func1
| | external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go:150
| | github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:227
| | github.com/cockroachdb/pebble.(*DB).collectTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:119
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (6) block 3468620/587: × checksum mismatch 49e1ed5c != d1b3a80d
| Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Fatal: local corruption detected: pebble: file 198033: block 3468620/587: × checksum mismatch 49e1ed5c != d1b3a80d
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:598
| | [...repeated from below...]
| Wraps: (3) pebble: file 198033
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:206
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:596
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:531
| | github.com/cockroachdb/pebble/sstable.(*Reader).readRangeDelBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:393
| | github.com/cockroachdb/pebble/sstable.(*Reader).NewRawRangeDelIter
| | external/com_github_cockroachdb_pebble/sstable/reader.go:253
| | github.com/cockroachdb/pebble.newCombinedDeletionKeyspanIter
| | external/com_github_cockroachdb_pebble/table_stats.go:982
| | github.com/cockroachdb/pebble.(*DB).loadTableRangeDelStats
| | external/com_github_cockroachdb_pebble/table_stats.go:384
| | github.com/cockroachdb/pebble.(*DB).loadTableStats.func1
| | external/com_github_cockroachdb_pebble/table_stats.go:335
| | github.com/cockroachdb/pebble.(*fileCacheHandle).withReader
| | external/com_github_cockroachdb_pebble/file_cache.go:381
| | github.com/cockroachdb/pebble.(*DB).loadTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:308
| | github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats-range1
| | external/com_github_cockroachdb_pebble/table_stats.go:285
| | github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats.(*LevelMetadata).All.func1
| | external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go:150
| | github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:227
| | github.com/cockroachdb/pebble.(*DB).collectTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:119
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
...
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[external/com_github_cockroachdb_pebble/table_stats.go#L118-L120](external/com_github_cockroachdb_pebble/table_stats.go#L118-L120)
[external/com_github_cockroachdb_pebble/table_stats.go#L226-L228](external/com_github_cockroachdb_pebble/table_stats.go#L226-L228)
[external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go#L149-L151](external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go#L149-L151)
[external/com_github_cockroachdb_pebble/table_stats.go#L284-L286](external/com_github_cockroachdb_pebble/table_stats.go#L284-L286)
[external/com_github_cockroachdb_pebble/table_stats.go#L307-L309](external/com_github_cockroachdb_pebble/table_stats.go#L307-L309)
[external/com_github_cockroachdb_pebble/file_cache.go#L380-L382](external/com_github_cockroachdb_pebble/file_cache.go#L380-L382)
[external/com_github_cockroachdb_pebble/table_stats.go#L334-L336](external/com_github_cockroachdb_pebble/table_stats.go#L334-L336)
[external/com_github_cockroachdb_pebble/table_stats.go#L383-L385](external/com_github_cockroachdb_pebble/table_stats.go#L383-L385)
[external/com_github_cockroachdb_pebble/table_stats.go#L981-L983](external/com_github_cockroachdb_pebble/table_stats.go#L981-L983)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L252-L254](external/com_github_cockroachdb_pebble/sstable/reader.go#L252-L254)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L392-L394](external/com_github_cockroachdb_pebble/sstable/reader.go#L392-L394)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L533-L535](external/com_github_cockroachdb_pebble/sstable/block/block.go#L533-L535)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L434-L436](external/com_github_cockroachdb_pebble/sstable/block/block.go#L434-L436)
[external/com_github_cockroachdb_pebble/event.go#L1332-L1334](external/com_github_cockroachdb_pebble/event.go#L1332-L1334)
[external/com_github_cockroachdb_pebble/event.go#L1373-L1375](external/com_github_cockroachdb_pebble/event.go#L1373-L1375)
[external/com_github_cockroachdb_pebble/event.go#L1173-L1175](external/com_github_cockroachdb_pebble/event.go#L1173-L1175)
[pkg/storage/pebble.go#L1230-L1232](pkg/storage/pebble.go#L1230-L1232)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).collectTableStats at line 119
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats at line 227
external/com_github_cockroachdb_pebble/internal/manifest/level_metadata.go in github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats.(*LevelMetadata).All.func1 at line 150
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats-range1 at line 285
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadTableStats at line 308
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).withReader at line 381
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadTableStats.func1 at line 335
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).loadTableRangeDelStats at line 384
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.newCombinedDeletionKeyspanIter at line 982
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).NewRawRangeDelIter at line 253
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readRangeDelBlock at line 393
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 534
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 435
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1333
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1374
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1174
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1231
```
### Tags
| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.3.4 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.3.4 |
| Cockroach SHA | 1cddb3a22695e967197844b214115eb645473efe |
| # of CPUs | 2 |
| # of Goroutines | 146 |
Jira issue: CRDB-63682
Contributor guide
Assessment
This issue has not been assessed yet.