Sentry: error.go:30: pebble: invalid internal key kind: INVALID (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/cockroach...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6938126836/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6938126836/?referrer=webhooks_plugin)

Panic Message:

```
error.go:30: pebble: invalid internal key kind: INVALID
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).destroy
| github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:1040
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Close
| github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:362
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Close
| github.com/cockroachdb/cockroach/pkg/storage/intent_interleaving_iter.go:1072
| [...repeated from below...]
Wraps: (2) forced error mark
| ×
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/error.go:30
| github.com/cockroachdb/pebble.(*Iterator).findNextEntry
| github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go:641
| github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
| github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go:1355
| github.com/cockroachdb/pebble.(*Iterator).SeekGE
| github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go:1251
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:375
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).SeekGE
| github.com/cockroachdb/cockroach/pkg/storage/intent_interleaving_iter.go:552
| github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:7211
| github.com/cockroachdb/cockroach/pkg/storage.ComputeStats
| github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:7176
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.ComputeRaftLogSize
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:2679
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftLogQueue).process
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go:694
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:984
| github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
| github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:977
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:927
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (4) pebble: invalid internal key kind: INVALID
Error types: (1) *withstack.withStack (2) *markers.withMark (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: pebble: invalid internal key kind: INVALID
error.go:30: *withstack.withStack (top exception)
*markers.withMark
panic.go:770: *withstack.withStack (1)
(check the extra data payloads)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L926-L928](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L926-L928)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L976-L978](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L976-L978)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/timeutil/timeout.go#L28-L30](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/timeutil/timeout.go#L28-L30)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L983-L985](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L983-L985)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go#L693-L695](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go#L693-L695)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L2678-L2680](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L2678-L2680)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7175-L7177](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7175-L7177)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7211-L7213](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7211-L7213)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/intent_interleaving_iter.go#L1071-L1073](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/intent_interleaving_iter.go#L1071-L1073)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/pebble_iterator.go#L361-L363](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/pebble_iterator.go#L361-L363)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/pebble_iterator.go#L1039-L1041](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/pebble_iterator.go#L1039-L1041)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L926-L928](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L926-L928)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L976-L978](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L976-L978)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/timeutil/timeout.go#L28-L30](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/timeutil/timeout.go#L28-L30)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L983-L985](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/queue.go#L983-L985)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go#L693-L695](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go#L693-L695)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L2678-L2680](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L2678-L2680)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7175-L7177](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7175-L7177)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7210-L7212](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/mvcc.go#L7210-L7212)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/intent_interleaving_iter.go#L551-L553](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/intent_interleaving_iter.go#L551-L553)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/pebble_iterator.go#L374-L376](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/storage/pebble_iterator.go#L374-L376)
[github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1250-L1252](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1250-L1252)
[github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1354-L1356](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1354-L1356)
[github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L640-L642](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L640-L642)
[github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/error.go#L29-L31](github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/error.go#L29-L31)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1 at line 927
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica at line 977
pkg/util/timeutil/timeout.go in pkg/util/timeutil.RunWithTimeout at line 29
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica.func1 at line 984
pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go in pkg/kv/kvserver.(*raftLogQueue).process at line 694
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.ComputeRaftLogSize at line 2679
pkg/storage/mvcc.go in pkg/storage.ComputeStats at line 7176
pkg/storage/mvcc.go in pkg/storage.ComputeStatsWithVisitors at line 7212
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).Close at line 1072
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).Close at line 362
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).destroy at line 1040
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1 at line 927
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica at line 977
pkg/util/timeutil/timeout.go in pkg/util/timeutil.RunWithTimeout at line 29
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica.func1 at line 984
pkg/kv/kvserver/pkg/kv/kvserver/raft_log_queue.go in pkg/kv/kvserver.(*raftLogQueue).process at line 694
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.ComputeRaftLogSize at line 2679
pkg/storage/mvcc.go in pkg/storage.ComputeStats at line 7176
pkg/storage/mvcc.go in pkg/storage.ComputeStatsWithVisitors at line 7211
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).SeekGE at line 552
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 375
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGE at line 1251
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit at line 1355
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).findNextEntry at line 641
github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/error.go in github.com/cockroachdb/pebble/internal/base.CorruptionErrorf at line 30
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.1.2 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.2 |
| Cockroach SHA | 7e81be6de75205c3d08b0d8dcc6ca188306abc27 |
| # of CPUs | 8 |
| # of Goroutines | 416 |

Jira issue: CRDB-55352

Start with pkg/storage/pebble_iterator.go and pkg/storage/intent_interleaving_iter.go, then trace the reported path through pkg/storage/mvcc.go, pkg/kv/kvserver/replica_raft.go, and raft_log_queue.go. Reproduce or narrow the invalid internal key failure from the Sentry report, and add a regression test or other verified fix once the cause is established.

Written by the indexing model from the issue text.