cockroachdb / cockroachdb/cockroach
Sentry: txn_coord_sender.go:1279: cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0}...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7630712795/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7630712795/?referrer=webhooks_plugin)
Panic Message:
```
txn_coord_sender.go:1279: cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0} lock=false stat=PENDING rts=1784890432.805371788,0 gul=1784890433.055371788,0 obs={n6@1784890432.805371788,0} already performed reads
(1) plan gist: AgHsAQIASAIAAAUCDAHuAQIAgAQCAAAFAAwHBgYG
Wraps: (2) while executing: SELECT count(*), COALESCE(sum(_), _), (SELECT count(*) FROM _ WHERE _ = '_') FROM _ AS OF SYSTEM TIME with_max_staleness(_) WHERE _ = '_'
Wraps: (3)
Wraps: (4) tags: [n×,client=×,hostnossl=,user=×]
Wraps: (5) assertion failure
Wraps: (6)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).SetFixedTimestamp
| pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1279
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).SetFixedTimestamp
| pkg/kv/txn.go:1734
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).NegotiateAndSend
| pkg/kv/txn.go:1519
| github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.func1
| pkg/sql/row/kv_fetcher.go:93
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
| pkg/sql/row/kv_batch_fetcher.go:689
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
| pkg/sql/row/kv_batch_fetcher.go:979
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch
| pkg/sql/row/kv_batch_fetcher.go:846
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).nextKV
| pkg/sql/row/kv_fetcher.go:334
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
| pkg/sql/row/kv_fetcher.go:355
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch
| pkg/sql/colfetcher/cfetcher.go:785
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next
| pkg/sql/colfetcher/colbatch_scan.go:270
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next
| pkg/sql/colexec/colexecutils/cancel_checker.go:62
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| pkg/sql/colexec/colexecbase/simple_project.go:114
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*fnOp).Next
| pkg/sql/colexec/colexecbase/fn_op.go:28
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*orderedAggregator).Next
| pkg/sql/colexec/ordered_aggregator.go:205
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
| pkg/sql/colexecop/operator.go:476
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*BatchSchemaSubsetEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:223
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*bufferOp).advance
| pkg/sql/colexec/buffer.go:45
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*caseOp).Next
| pkg/sql/colexec/case.go:193
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:153
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.constInt64Op.Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go:379
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:153
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*castIdentityOp).Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go:1275
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| pkg/sql/colexec/colexecbase/simple_project.go:114
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
Wraps: (7) cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0} lock=false stat=PENDING rts=1784890432.805371788,0 gul=1784890433.055371788,0 obs={n6@1784890432.805371788,0} already performed reads
Error types: (1) *safedetails.withSafeDetails (2) *safedetails.withSafeDetails (3) *colexecerror.StorageError (4) *contexttags.withContext (5) *assert.withAssertionFailure (6) *errbase.opaqueWrapper (7) *errutil.leafError
-- report composition:
*errutil.leafError: cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0} lock=false stat=PENDING rts=1784890432.805371788,0 gul=1784890433.055371788,0 obs={n6@1784890432.805371788,0} already performed reads
txn_coord_sender.go:1279: *withstack.withStack (top exception)
*assert.withAssertionFailure
*contexttags.withContext: n6
*colexecerror.StorageError
*safedetails.withSafeDetails: while executing: SELECT count(*), COALESCE(sum(_), _), (SELECT count(*) FROM _ WHERE _ = '_') FROM _ AS OF SYSTEM TIME with_max_staleness(_) WHERE _ = '_'
*safedetails.withSafeDetails: plan gist: AgHsAQIASAIAAAUCDAHuAQIAgAQCAAAFAAwHBgYG
```
Stacktrace (expand for inline code snippets):
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/colexecbase/simple_project.go#L113-L115](pkg/sql/colexec/colexecbase/simple_project.go#L113-L115)
[bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L1274-L1276](bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L1274-L1276)
[pkg/sql/colexec/colexecutils/operator.go#L152-L154](pkg/sql/colexec/colexecutils/operator.go#L152-L154)
[bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go#L378-L380](bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go#L378-L380)
[pkg/sql/colexec/colexecutils/operator.go#L152-L154](pkg/sql/colexec/colexecutils/operator.go#L152-L154)
[pkg/sql/colexec/case.go#L192-L194](pkg/sql/colexec/case.go#L192-L194)
[pkg/sql/colexec/buffer.go#L44-L46](pkg/sql/colexec/buffer.go#L44-L46)
[pkg/sql/colexec/colexecutils/operator.go#L222-L224](pkg/sql/colexec/colexecutils/operator.go#L222-L224)
[pkg/sql/colexecop/operator.go#L475-L477](pkg/sql/colexecop/operator.go#L475-L477)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/ordered_aggregator.go#L204-L206](pkg/sql/colexec/ordered_aggregator.go#L204-L206)
[pkg/sql/colexec/colexecbase/fn_op.go#L27-L29](pkg/sql/colexec/colexecbase/fn_op.go#L27-L29)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/colexecbase/simple_project.go#L113-L115](pkg/sql/colexec/colexecbase/simple_project.go#L113-L115)
[pkg/sql/colexec/colexecutils/cancel_checker.go#L61-L63](pkg/sql/colexec/colexecutils/cancel_checker.go#L61-L63)
[pkg/sql/colfetcher/colbatch_scan.go#L269-L271](pkg/sql/colfetcher/colbatch_scan.go#L269-L271)
[pkg/sql/colfetcher/cfetcher.go#L784-L786](pkg/sql/colfetcher/cfetcher.go#L784-L786)
[pkg/sql/row/kv_fetcher.go#L354-L356](pkg/sql/row/kv_fetcher.go#L354-L356)
[pkg/sql/row/kv_fetcher.go#L333-L335](pkg/sql/row/kv_fetcher.go#L333-L335)
[pkg/sql/row/kv_batch_fetcher.go#L845-L847](pkg/sql/row/kv_batch_fetcher.go#L845-L847)
[pkg/sql/row/kv_batch_fetcher.go#L978-L980](pkg/sql/row/kv_batch_fetcher.go#L978-L980)
[pkg/sql/row/kv_batch_fetcher.go#L688-L690](pkg/sql/row/kv_batch_fetcher.go#L688-L690)
[pkg/sql/row/kv_fetcher.go#L92-L94](pkg/sql/row/kv_fetcher.go#L92-L94)
[pkg/kv/txn.go#L1518-L1520](pkg/kv/txn.go#L1518-L1520)
[pkg/kv/txn.go#L1733-L1735](pkg/kv/txn.go#L1733-L1735)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L1278-L1280](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L1278-L1280)
```
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 114
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go in pkg/sql/colexec/colexecbase.(*castIdentityOp).Next at line 1275
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 153
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go in pkg/sql/colexec/colexecbase.constInt64Op.Next at line 379
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 153
pkg/sql/colexec/case.go in pkg/sql/colexec.(*caseOp).Next at line 193
pkg/sql/colexec/buffer.go in pkg/sql/colexec.(*bufferOp).advance at line 45
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*BatchSchemaSubsetEnforcer).Next at line 223
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 476
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/ordered_aggregator.go in pkg/sql/colexec.(*orderedAggregator).Next at line 205
pkg/sql/colexec/colexecbase/fn_op.go in pkg/sql/colexec/colexecbase.(*fnOp).Next at line 28
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 114
pkg/sql/colexec/colexecutils/cancel_checker.go in pkg/sql/colexec/colexecutils.(*CancelChecker).Next at line 62
pkg/sql/colfetcher/colbatch_scan.go in pkg/sql/colfetcher.(*ColBatchScan).Next at line 270
pkg/sql/colfetcher/cfetcher.go in pkg/sql/colfetcher.(*cFetcher).NextBatch at line 785
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).NextKV at line 355
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).nextKV at line 334
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).NextBatch at line 846
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).nextBatch at line 979
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).fetch at line 689
pkg/sql/row/kv_fetcher.go in pkg/sql/row.newTxnKVFetcher.func1 at line 93
pkg/kv/txn.go in pkg/kv.(*Txn).NegotiateAndSend at line 1519
pkg/kv/txn.go in pkg/kv.(*Txn).SetFixedTimestamp at line 1734
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).SetFixedTimestamp at line 1279
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.2.3 |
| Go Version | go1.25.5 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.2.3 |
| Cockroach SHA | 90d3b6080727dc810224a99903cb2e88b81e91ae |
| # of CPUs | 2 |
| # of Goroutines | 782 |
Contributor guide
Research direction
Start at pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1279 and trace SetFixedTimestamp through pkg/kv/txn.go and pkg/sql/row/kv_fetcher.go. Investigate the reported AS OF SYSTEM TIME query and the transaction state showing reads before the fixed timestamp was set. Done means the assertion no longer occurs for this execution path and a regression test covers the failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100