cockroachdb / cockroachdb/cockroach

Sentry: crash_reporting.go:432: command writing below closed timestamp; cmd: 213a749429ced1ee, write ts: 1779901628.973484406,0, batch state closed: 1779901629.956152670,0, command closed: 1779901626....

Open
#171,025 3 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3 C-bug O-sentry
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7509641443/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7509641443/?referrer=webhooks_plugin)

Panic Message:

```
crash_reporting.go:432: command writing below closed timestamp; cmd: 213a749429ced1ee, write ts: 1779901628.973484406,0, batch state closed: 1779901629.956152670,0, command closed: 1779901626.147974286,0, request: ConditionalPut [/Table/54/1/×/×/×/×], [txn: e75407bd], lease: repl=(n1,s1):1 seq=1 start=0,0 epo=1 min-exp=0,0 pro=1779901632.932392154,0 acq=Request.
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/util/log/logcrash.ReportOrPanic
| pkg/util/log/logcrash/crash_reporting.go:432
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoWriteBelowClosedTimestamp
| pkg/kv/kvserver/replica_app_batch.go:743
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| pkg/kv/kvserver/replica_app_batch.go:119
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| pkg/kv/kvserver/apply/cmd.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| pkg/kv/kvserver/apply/task.go:279
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| pkg/kv/kvserver/apply/task.go:246
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1269
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:846
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:682
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:505
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
| command writing below closed timestamp; cmd: 213a749429ced1ee, write ts: 1779901628.973484406,0, batch state closed: 1779901629.956152670,0, command closed: 1779901626.147974286,0, request: ConditionalPut [/Table/54/1/×/×/×/×], [txn: e75407bd], lease: repl=(n1,s1):1 seq=1 start=0,0 epo=1 min-exp=0,0 pro=1779901632.932392154,0 acq=Request.
| (1) assertion failure
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).assertNoWriteBelowClosedTimestamp
| | pkg/kv/kvserver/replica_app_batch.go:737
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage
| | pkg/kv/kvserver/replica_app_batch.go:119
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter
| | pkg/kv/kvserver/apply/cmd.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | pkg/kv/kvserver/apply/task.go:279
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | pkg/kv/kvserver/apply/task.go:246
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | pkg/kv/kvserver/replica_raft.go:1269
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | pkg/kv/kvserver/replica_raft.go:846
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | pkg/kv/kvserver/store_raft.go:682
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:419
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:319
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:505
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (3) command writing below closed timestamp; cmd: 213a749429ced1ee, write ts: 1779901628.973484406,0, batch state closed: 1779901629.956152670,0, command closed: 1779901626.147974286,0, request: ConditionalPut [/Table/54/1/×/×/×/×], [txn: e75407bd], lease: repl=(n1,s1):1 seq=1 start=0,0 epo=1 min-exp=0,0 pro=1779901632.932392154,0 acq=Request.
| Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
Wraps: (3) command writing below closed timestamp; cmd: 213a749429ced1ee, write ts: 1779901628.973484406,0, batch state closed: 1779901629.956152670,0, command closed: 1779901626.147974286,0, request: ConditionalPut [/Table/54/1/×/×/×/×], [txn: e75407bd], lease: repl=(n1,s1):1 seq=1 start=0,0 epo=1 min-exp=0,0 pro=1779901632.932392154,0 acq=Request.
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: command writing below closed timestamp; cmd: 213a749429ced1ee, write ts: 1779901628.973484406,0, batch state closed: 1779901629.956152670,0, command closed: 1779901626.147974286,0, request: ConditionalPut [/Table/54/1/×/×/×/×], [txn: e75407bd], lease: repl=(n1,s1):1 seq=1 start=0,0 epo=1 min-exp=0,0 pro=1779901632.932392154,0 acq=Request.
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
crash_reporting.go:432: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/stop/stopper.go#L504-L506](pkg/util/stop/stopper.go#L504-L506)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L681-L683](pkg/kv/kvserver/store_raft.go#L681-L683)
[pkg/kv/kvserver/replica_raft.go#L845-L847](pkg/kv/kvserver/replica_raft.go#L845-L847)
[pkg/kv/kvserver/replica_raft.go#L1268-L1270](pkg/kv/kvserver/replica_raft.go#L1268-L1270)
[pkg/kv/kvserver/apply/task.go#L245-L247](pkg/kv/kvserver/apply/task.go#L245-L247)
[pkg/kv/kvserver/apply/task.go#L278-L280](pkg/kv/kvserver/apply/task.go#L278-L280)
[pkg/kv/kvserver/apply/cmd.go#L186-L188](pkg/kv/kvserver/apply/cmd.go#L186-L188)
[pkg/kv/kvserver/replica_app_batch.go#L118-L120](pkg/kv/kvserver/replica_app_batch.go#L118-L120)
[pkg/kv/kvserver/replica_app_batch.go#L742-L744](pkg/kv/kvserver/replica_app_batch.go#L742-L744)
[pkg/util/log/logcrash/crash_reporting.go#L431-L433](pkg/util/log/logcrash/crash_reporting.go#L431-L433)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 505
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 682
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 846
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1269
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 246
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 279
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCmdIter at line 187
pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).Stage at line 119
pkg/kv/kvserver/replica_app_batch.go in pkg/kv/kvserver.(*replicaAppBatch).assertNoWriteBelowClosedTimestamp at line 743
pkg/util/log/logcrash/crash_reporting.go in pkg/util/log/logcrash.ReportOrPanic at line 432
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v24.3.33 |
| Go Version | go1.22.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.3.33 |
| Cockroach SHA | 4553626be4d29aff5517b42da4113363d64bc030 |
| # of CPUs | 32 |
| # of Goroutines | 644 |

Jira issue: CRDB-64285

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.