cockroachdb / cockroachdb/cockroach

Sentry: crash_reporting.go:415: raft closed timestamp regression in cmd: 16b2787ab44a2bb7 (term: 8, index: 54); batch state: 1763571500.814668693,0, command: 1763571497.497236238,0, lease: repl=(n1,s1...

Open
#158,059 1 comment 0 reactions 0 assignees View on GitHub
branch-release-24.1 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7044530846/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7044530846/?referrer=webhooks_plugin)

Panic Message:

```
crash_reporting.go:415: raft closed timestamp regression in cmd: 16b2787ab44a2bb7 (term: 8, index: 54); batch state: 1763571500.814668693,0, command: 1763571497.497236238,0, lease: repl=(n1,s1):1 seq=3 start=1763571502.019815715,0 epo=2 pro=1763571502.280832177,0, req: Put [/System/StatusNode/2], applying at LAI: 19.
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
| github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:415
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:788
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:122
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1098
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| raft closed timestamp regression in cmd: 16b2787ab44a2bb7 (term: 8, index: 54); batch state: 1763571500.814668693,0, command: 1763571497.497236238,0, lease: repl=(n1,s1):1 seq=3 start=1763571502.019815715,0 epo=2 pro=1763571502.280832177,0, req: Put [/System/StatusNode/2], applying at LAI: 19.
| (1) assertion failure
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:781
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:122
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1098
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:757
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) raft closed timestamp regression in cmd: 16b2787ab44a2bb7 (term: 8, index: 54); batch state: 1763571500.814668693,0, command: 1763571497.497236238,0, lease: repl=(n1,s1):1 seq=3 start=1763571502.019815715,0 epo=2 pro=1763571502.280832177,0, req: Put [/System/StatusNode/2], applying at LAI: 19.
| | Closed timestamp was set by req: under lease: ; applied at LAI: 0. Batch idx: 0.
| | Raft log tail:
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| | ×
| | ×
| | ×
| |
| |
| | ×
| | ×
| | ×
| |
| | ×
| | ×
| | ×
| | ×
| Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
Wraps: (3) raft closed timestamp regression in cmd: 16b2787ab44a2bb7 (term: 8, index: 54); batch state: 1763571500.814668693,0, command: 1763571497.497236238,0, lease: repl=(n1,s1):1 seq=3 start=1763571502.019815715,0 epo=2 pro=1763571502.280832177,0, req: Put [/System/StatusNode/2], applying at LAI: 19.
| Closed timestamp was set by req: under lease: ; applied at LAI: 0. Batch idx: 0.
| Raft log tail:
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
| ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: raft close...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/stop/stopper.go#L484-L486)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L320-L322](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L320-L322)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L417-L419](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L417-L419)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L688-L690](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L688-L690)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L756-L758](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L756-L758)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1097-L1099](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1097-L1099)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/apply/task.go#L250-L252](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/apply/task.go#L250-L252)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/apply/task.go#L283-L285](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/apply/task.go#L283-L285)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/apply/cmd.go#L191-L193](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/apply/cmd.go#L191-L193)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L121-L123](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L121-L123)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L787-L789](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go#L787-L789)
[https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/log/logcrash/crash_reporting.go#L414-L416](https://github.com/cockroachdb/cockroach/blob/7e81be6de75205c3d08b0d8dcc6ca188306abc27/pkg/util/log/logcrash/crash_reporting.go#L414-L416)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 321
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 418
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 689
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 757
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1098
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 251
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 284
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCmdIter at line 192
pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).Stage at line 122
pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).assertNoCmdClosedTimestampRegression at line 788
pkg/util/log/logcrash/crash_reporting.go in pkg/util/log/logcrash.ReportOrPanic at line 415
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.1.2 |
| Go Version | go1.22.3 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.1.2 |
| Cockroach SHA | 7e81be6de75205c3d08b0d8dcc6ca188306abc27 |
| # of CPUs | 12 |
| # of Goroutines | 429 |

Jira issue: CRDB-57082

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.