cockroachdb / cockroachdb/cockroach

Sentry: crash_reporting.go:410: raft closed timestamp regression in cmd: 45da41ee9218c03e (term: 8, index: 46193); batch state: 1757225375.096619782,0, command: 1757224838.944929847,0, lease: repl=(n1...

Open
#153,123 2 comments 0 reactions 0 assignees View on GitHub
branch-release-24.1 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6862500869/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6862500869/?referrer=webhooks_plugin)

Panic Message:

```
crash_reporting.go:410: raft closed timestamp regression in cmd: 45da41ee9218c03e (term: 8, index: 46193); batch state: 1757225375.096619782,0, command: 1757224838.944929847,0, lease: repl=(n1,s1):1 seq=2 start=1756954526.423065033,0 exp=1757225384.096619782,9 pro=1757225378.096619782,9, req: Put [/Local/Range/Min/×/×], applying at LAI: 475.
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
| github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:410
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:784
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:118
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:279
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:246
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1094
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:753
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:684
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:412
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:315
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| raft closed timestamp regression in cmd: 45da41ee9218c03e (term: 8, index: 46193); batch state: 1757225375.096619782,0, command: 1757224838.944929847,0, lease: repl=(n1,s1):1 seq=2 start=1756954526.423065033,0 exp=1757225384.096619782,9 pro=1757225378.096619782,9, req: Put [/Local/Range/Min/×/×], applying at LAI: 475.
| (1) assertion failure
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:777
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:118
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:279
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1094
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:753
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:684
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:412
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:315
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) raft closed timestamp regression in cmd: 45da41ee9218c03e (term: 8, index: 46193); batch state: 1757225375.096619782,0, command: 1757224838.944929847,0, lease: repl=(n1,s1):1 seq=2 start=1756954526.423065033,0 exp=1757225384.096619782,9 pro=1757225378.096619782,9, req: Put [/Local/Range/Min/×/×], applying at LAI: 475.
| | Closed timestamp was set by req: under lease: repl=(n1,s1):1 seq=2 start=1756954526.423065033,0 exp=1757225384.096619782,9 pro=1757225378.096619782,9; applied at LAI: 474. Batch idx: 0.
| | Raft log tail:
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/util/stop/stopper.go#L479-L481](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/util/stop/stopper.go#L479-L481)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L314-L316](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L314-L316)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L411-L413](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L411-L413)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L683-L685](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L683-L685)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L752-L754](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L752-L754)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1093-L1095](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1093-L1095)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/apply/task.go#L245-L247](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/apply/task.go#L245-L247)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/apply/task.go#L278-L280](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/apply/task.go#L278-L280)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/apply/cmd.go#L186-L188](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/apply/cmd.go#L186-L188)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L117-L119](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L117-L119)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L783-L785](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L783-L785)
[https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/util/log/logcrash/crash_reporting.go#L409-L411](https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/util/log/logcrash/crash_reporting.go#L409-L411)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 480
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 315
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 412
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 684
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 753
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1094
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 246
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 279
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCmdIter at line 187
pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).Stage at line 118
pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression at line 784
pkg/util/log/logcrash/crash_reporting.go in pkg/util/log/logcrash.ReportOrPanic at line 410
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.14 |
| Go Version | go1.22.5 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.14 |
| Cockroach SHA | c8459c085e2258f589949e970a527a0342871276 |
| # of CPUs | 64 |
| # of Goroutines | 791 |

Jira issue: CRDB-54217

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.