cockroachdb / cockroachdb/cockroach
Sentry: txn_interceptor_span_refresher.go:388: txn status not finalized after successful retried EndTxn: "sql txn" meta={id=52e7a534 key=/Table/1151146/180/×/× iso=Serializable pri=0.00323029 epo=0 ...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7392063000/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7392063000/?referrer=webhooks_plugin)
Panic Message:
```
txn_interceptor_span_refresher.go:388: txn status not finalized after successful retried EndTxn: "sql txn" meta={id=52e7a534 key=/Table/1151146/180/×/× iso=Serializable pri=0.00323029 epo=0 ts=1775540210.445055935,1 min=1775540196.683500930,0 seq=55} lock=true stat=PREPARED rts=1775540210.445055935,1 gul=1775540196.933500930,0 obs={n1@1775540202.620055172,0 n3@1775540202.577124890,0 n4@1775540201.177633243,0 n6@1775540202.604878379,0 n7@1775540202.560992586,0, ...} int=55
(1) assertion failure
Wraps: (2)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).splitEndTxnAndRetrySend
| pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:388
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).maybeRefreshAndRetrySend
| pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:300
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
| pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:238
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:163
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:190
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:340
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go:267
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:113
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:266
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
| pkg/kv/kvclient/kvcoord/txn_coord_sender.go:571
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
| pkg/kv/db.go:1167
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
| pkg/kv/txn.go:1372
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Prepare
| pkg/kv/txn.go:1078
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPrepareTransactionInOpenStateInternal
| pkg/sql/two_phase_commit.go:116
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPrepareTransactionInOpenState
| pkg/sql/two_phase_commit.go:40
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| pkg/sql/conn_executor_exec.go:848
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func2
| pkg/sql/conn_executor_exec.go:175
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
| pkg/sql/conn_executor_exec.go:4429
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| pkg/sql/conn_executor_exec.go:174
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1
| pkg/sql/conn_executor.go:2378
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| pkg/sql/conn_executor.go:2383
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| pkg/sql/conn_executor.go:2291
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| pkg/sql/conn_executor.go:1067
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands
| pkg/sql/pgwire/conn.go:253
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func4
| pkg/sql/pgwire/server.go:1228
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (3) txn status not finalized after successful retried EndTxn: "sql txn" meta={id=52e7a534 key=/Table/1151146/180/×/× iso=Serializable pri=0.00323029 epo=0 ts=1775540210.445055935,1 min=1775540196.683500930,0 seq=55} lock=true stat=PREPARED rts=1775540210.445055935,1 gul=1775540196.933500930,0 obs={n1@1775540202.620055172,0 n3@1775540202.577124890,0 n4@1775540201.177633243,0 n6@1775540202.604878379,0 n7@1775540202.560992586,0, ...} int=55
Error types: (1) *assert.withAssertionFailure (2) *errbase.opaqueWrapper (3) *errutil.leafError
-- report composition:
*errutil.leafError: txn status not finalized after successful retried EndTxn: "sql txn" meta={id=52e7a534 key=/Table/1151146/180/×/× iso=Serializable pri=0.00323029 epo=0 ts=1775540210.445055935,1 min=1775540196.683500930,0 seq=55} lock=true stat=PREPARED rts=1775540210.445055935,1 gul=1775540196.933500930,0 obs={n1@1775540202.620055172,0 n3@1775540202.577124890,0 n4@1775540201.177633243,0 n6@1775540202.604878379,0 n7@1775540202.560992586,0, ...} int=55
txn_interceptor_span_refresher.go:388: *withstack.withStack (top exception)
*assert.withAssertionFailure
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/sql/pgwire/server.go#L1227-L1229](pkg/sql/pgwire/server.go#L1227-L1229)
[pkg/sql/pgwire/conn.go#L252-L254](pkg/sql/pgwire/conn.go#L252-L254)
[pkg/sql/conn_executor.go#L1066-L1068](pkg/sql/conn_executor.go#L1066-L1068)
[pkg/sql/conn_executor.go#L2290-L2292](pkg/sql/conn_executor.go#L2290-L2292)
[pkg/sql/conn_executor.go#L2382-L2384](pkg/sql/conn_executor.go#L2382-L2384)
[pkg/sql/conn_executor.go#L2377-L2379](pkg/sql/conn_executor.go#L2377-L2379)
[pkg/sql/conn_executor_exec.go#L173-L175](pkg/sql/conn_executor_exec.go#L173-L175)
[pkg/sql/conn_executor_exec.go#L4428-L4430](pkg/sql/conn_executor_exec.go#L4428-L4430)
[pkg/sql/conn_executor_exec.go#L174-L176](pkg/sql/conn_executor_exec.go#L174-L176)
[pkg/sql/conn_executor_exec.go#L847-L849](pkg/sql/conn_executor_exec.go#L847-L849)
[pkg/sql/two_phase_commit.go#L39-L41](pkg/sql/two_phase_commit.go#L39-L41)
[pkg/sql/two_phase_commit.go#L115-L117](pkg/sql/two_phase_commit.go#L115-L117)
[pkg/kv/txn.go#L1077-L1079](pkg/kv/txn.go#L1077-L1079)
[pkg/kv/txn.go#L1371-L1373](pkg/kv/txn.go#L1371-L1373)
[pkg/kv/db.go#L1166-L1168](pkg/kv/db.go#L1166-L1168)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L570-L572](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L570-L572)
[pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L265-L267](pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L265-L267)
[pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L112-L114](pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L112-L114)
[pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go#L266-L268](pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go#L266-L268)
[pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L339-L341](pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L339-L341)
[pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L189-L191](pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L189-L191)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L162-L164](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L162-L164)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L237-L239](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L237-L239)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L299-L301](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L299-L301)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L387-L389](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L387-L389)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/sql/pgwire/server.go in pkg/sql/pgwire.(*Server).serveImpl.func4 at line 1228
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommands at line 253
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 1067
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 2291
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2383
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func1 at line 2378
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 174
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 4429
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func2 at line 175
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 848
pkg/sql/two_phase_commit.go in pkg/sql.(*connExecutor).execPrepareTransactionInOpenState at line 40
pkg/sql/two_phase_commit.go in pkg/sql.(*connExecutor).execPrepareTransactionInOpenStateInternal at line 116
pkg/kv/txn.go in pkg/kv.(*Txn).Prepare at line 1078
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1372
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1167
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 571
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go in pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked at line 266
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked at line 113
pkg/kv/kvclient/kvcoord/txn_interceptor_write_buffer.go in pkg/kv/kvclient/kvcoord.(*txnWriteBuffer).SendLocked at line 267
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked at line 340
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go in pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked at line 190
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked at line 163
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts at line 238
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).maybeRefreshAndRetrySend at line 300
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).splitEndTxnAndRetrySend at line 388
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.4.5 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.5 |
| Cockroach SHA | a5d3d359798dfbfa7a3cfd129441674afe498592 |
| # of CPUs | 32 |
| # of Goroutines | 9996 |
Jira issue: CRDB-62568
Contributor guide
Assessment
This issue has not been assessed yet.