Sentry: pebble.go:1481: log.Fatal: local corruption detected: start timestamp mismatch (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricE...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7563243369/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7563243369/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1481: log.Fatal: local corruption detected: start timestamp mismatch
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func1
| github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1481
| github.com/cockroachdb/pebble.TeeEventListener.func1
| github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/event.go:763
| github.com/cockroachdb/pebble.(*DB).flush.func1
| github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1406
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble.(*DB).flush
| github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1397
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| start timestamp mismatch
| (1) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/storage.(*MVCCValueMerger).Finish
| | github.com/cockroachdb/cockroach/pkg/storage/pebble_merge.go:298
| | github.com/cockroachdb/pebble.finishValueMerger
| | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merger.go:34
| | github.com/cockroachdb/pebble.(*compactionIter).Next
| | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction_iter.go:516
| | github.com/cockroachdb/pebble.(*DB).runCompaction
| | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:3245
| | github.com/cockroachdb/pebble.(*DB).flush1
| | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1656
| | github.com/cockroachdb/pebble.(*DB).flush.func1
| | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1404
| | runtime/pprof.Do
| | GOROOT/src/runtime/pprof/runtime.go:51
| | github.com/cockroachdb/pebble.(*DB).flush
| | github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1397
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) start timestamp mismatch
| Error types: (1) *markers.withMark (2) *withstack.withStack (3) *errutil.leafError
Wraps: (3) log.Fatal: local corruption detected: start timestamp mismatch
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: local corruption detected: start timestamp mismatch
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
pebble.go:1481: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go#L1396-L1398](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go#L1396-L1398)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go#L1405-L1407](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go#L1405-L1407)
[github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/event.go#L762-L764](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/event.go#L762-L764)
[https://github.com/cockroachdb/cockroach/blob/de8d079c03d6f67d574a303071b56f311cd56ba6/pkg/storage/pebble.go#L1480-L1482](https://github.com/cockroachdb/cockroach/blob/de8d079c03d6f67d574a303071b56f311cd56ba6/pkg/storage/pebble.go#L1480-L1482)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).flush at line 1397
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).flush.func1 at line 1406
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func1 at line 763
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func1 at line 1481
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.1.29 |
| Go Version | go1.22.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.29 |
| Cockroach SHA | de8d079c03d6f67d574a303071b56f311cd56ba6 |
| # of CPUs | 1 |
| # of Goroutines | 306 |

Jira issue: CRDB-64952

Start at pkg/storage/pebble.go:1481, then trace the reported error through pkg/storage/pebble_merge.go:298 and the Pebble compaction stack frames in the report. Determine why the start timestamp mismatch occurs during a flush and what change would prevent the local-corruption crash; the issue provides no reproduction or test to run.

Written by the indexing model from the issue text.