cockroachdb / cockroachdb/cockroach

Sentry: txn_coord_sender.go:1279: cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0}...

Open
#172,784 4 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7630712795/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7630712795/?referrer=webhooks_plugin)

Panic Message:

```
txn_coord_sender.go:1279: cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0} lock=false stat=PENDING rts=1784890432.805371788,0 gul=1784890433.055371788,0 obs={n6@1784890432.805371788,0} already performed reads
(1) plan gist: AgHsAQIASAIAAAUCDAHuAQIAgAQCAAAFAAwHBgYG
Wraps: (2) while executing: SELECT count(*), COALESCE(sum(_), _), (SELECT count(*) FROM _ WHERE _ = '_') FROM _ AS OF SYSTEM TIME with_max_staleness(_) WHERE _ = '_'
Wraps: (3)
Wraps: (4) tags: [n×,client=×,hostnossl=,user=×]
Wraps: (5) assertion failure
Wraps: (6)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).SetFixedTimestamp
| pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1279
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).SetFixedTimestamp
| pkg/kv/txn.go:1734
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).NegotiateAndSend
| pkg/kv/txn.go:1519
| github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.func1
| pkg/sql/row/kv_fetcher.go:93
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
| pkg/sql/row/kv_batch_fetcher.go:689
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
| pkg/sql/row/kv_batch_fetcher.go:979
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch
| pkg/sql/row/kv_batch_fetcher.go:846
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).nextKV
| pkg/sql/row/kv_fetcher.go:334
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
| pkg/sql/row/kv_fetcher.go:355
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch
| pkg/sql/colfetcher/cfetcher.go:785
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next
| pkg/sql/colfetcher/colbatch_scan.go:270
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next
| pkg/sql/colexec/colexecutils/cancel_checker.go:62
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| pkg/sql/colexec/colexecbase/simple_project.go:114
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*fnOp).Next
| pkg/sql/colexec/colexecbase/fn_op.go:28
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*orderedAggregator).Next
| pkg/sql/colexec/ordered_aggregator.go:205
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
| pkg/sql/colexecop/operator.go:476
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*BatchSchemaSubsetEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:223
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*bufferOp).advance
| pkg/sql/colexec/buffer.go:45
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*caseOp).Next
| pkg/sql/colexec/case.go:193
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:153
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.constInt64Op.Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go:379
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:153
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*castIdentityOp).Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go:1275
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| pkg/sql/colexec/colexecbase/simple_project.go:114
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
Wraps: (7) cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0} lock=false stat=PENDING rts=1784890432.805371788,0 gul=1784890433.055371788,0 obs={n6@1784890432.805371788,0} already performed reads
Error types: (1) *safedetails.withSafeDetails (2) *safedetails.withSafeDetails (3) *colexecerror.StorageError (4) *contexttags.withContext (5) *assert.withAssertionFailure (6) *errbase.opaqueWrapper (7) *errutil.leafError
-- report composition:
*errutil.leafError: cannot set fixed timestamp, txn "sql txn" meta={id=a49574fb key=/Min iso=Serializable pri=0.02657315 epo=0 ts=1784890432.805371788,0 min=1784890432.805371788,0 seq=0} lock=false stat=PENDING rts=1784890432.805371788,0 gul=1784890433.055371788,0 obs={n6@1784890432.805371788,0} already performed reads
txn_coord_sender.go:1279: *withstack.withStack (top exception)
*assert.withAssertionFailure
*contexttags.withContext: n6
*colexecerror.StorageError
*safedetails.withSafeDetails: while executing: SELECT count(*), COALESCE(sum(_), _), (SELECT count(*) FROM _ WHERE _ = '_') FROM _ AS OF SYSTEM TIME with_max_staleness(_) WHERE _ = '_'
*safedetails.withSafeDetails: plan gist: AgHsAQIASAIAAAUCDAHuAQIAgAQCAAAFAAwHBgYG
```

Stacktrace (expand for inline code snippets):

[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/colexecbase/simple_project.go#L113-L115](pkg/sql/colexec/colexecbase/simple_project.go#L113-L115)
[bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L1274-L1276](bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L1274-L1276)
[pkg/sql/colexec/colexecutils/operator.go#L152-L154](pkg/sql/colexec/colexecutils/operator.go#L152-L154)
[bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go#L378-L380](bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go#L378-L380)
[pkg/sql/colexec/colexecutils/operator.go#L152-L154](pkg/sql/colexec/colexecutils/operator.go#L152-L154)
[pkg/sql/colexec/case.go#L192-L194](pkg/sql/colexec/case.go#L192-L194)
[pkg/sql/colexec/buffer.go#L44-L46](pkg/sql/colexec/buffer.go#L44-L46)
[pkg/sql/colexec/colexecutils/operator.go#L222-L224](pkg/sql/colexec/colexecutils/operator.go#L222-L224)
[pkg/sql/colexecop/operator.go#L475-L477](pkg/sql/colexecop/operator.go#L475-L477)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/ordered_aggregator.go#L204-L206](pkg/sql/colexec/ordered_aggregator.go#L204-L206)
[pkg/sql/colexec/colexecbase/fn_op.go#L27-L29](pkg/sql/colexec/colexecbase/fn_op.go#L27-L29)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/colexecbase/simple_project.go#L113-L115](pkg/sql/colexec/colexecbase/simple_project.go#L113-L115)
[pkg/sql/colexec/colexecutils/cancel_checker.go#L61-L63](pkg/sql/colexec/colexecutils/cancel_checker.go#L61-L63)
[pkg/sql/colfetcher/colbatch_scan.go#L269-L271](pkg/sql/colfetcher/colbatch_scan.go#L269-L271)
[pkg/sql/colfetcher/cfetcher.go#L784-L786](pkg/sql/colfetcher/cfetcher.go#L784-L786)
[pkg/sql/row/kv_fetcher.go#L354-L356](pkg/sql/row/kv_fetcher.go#L354-L356)
[pkg/sql/row/kv_fetcher.go#L333-L335](pkg/sql/row/kv_fetcher.go#L333-L335)
[pkg/sql/row/kv_batch_fetcher.go#L845-L847](pkg/sql/row/kv_batch_fetcher.go#L845-L847)
[pkg/sql/row/kv_batch_fetcher.go#L978-L980](pkg/sql/row/kv_batch_fetcher.go#L978-L980)
[pkg/sql/row/kv_batch_fetcher.go#L688-L690](pkg/sql/row/kv_batch_fetcher.go#L688-L690)
[pkg/sql/row/kv_fetcher.go#L92-L94](pkg/sql/row/kv_fetcher.go#L92-L94)
[pkg/kv/txn.go#L1518-L1520](pkg/kv/txn.go#L1518-L1520)
[pkg/kv/txn.go#L1733-L1735](pkg/kv/txn.go#L1733-L1735)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L1278-L1280](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L1278-L1280)

```
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 114
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go in pkg/sql/colexec/colexecbase.(*castIdentityOp).Next at line 1275
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 153
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go in pkg/sql/colexec/colexecbase.constInt64Op.Next at line 379
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 153
pkg/sql/colexec/case.go in pkg/sql/colexec.(*caseOp).Next at line 193
pkg/sql/colexec/buffer.go in pkg/sql/colexec.(*bufferOp).advance at line 45
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*BatchSchemaSubsetEnforcer).Next at line 223
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 476
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/ordered_aggregator.go in pkg/sql/colexec.(*orderedAggregator).Next at line 205
pkg/sql/colexec/colexecbase/fn_op.go in pkg/sql/colexec/colexecbase.(*fnOp).Next at line 28
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 114
pkg/sql/colexec/colexecutils/cancel_checker.go in pkg/sql/colexec/colexecutils.(*CancelChecker).Next at line 62
pkg/sql/colfetcher/colbatch_scan.go in pkg/sql/colfetcher.(*ColBatchScan).Next at line 270
pkg/sql/colfetcher/cfetcher.go in pkg/sql/colfetcher.(*cFetcher).NextBatch at line 785
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).NextKV at line 355
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).nextKV at line 334
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).NextBatch at line 846
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).nextBatch at line 979
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).fetch at line 689
pkg/sql/row/kv_fetcher.go in pkg/sql/row.newTxnKVFetcher.func1 at line 93
pkg/kv/txn.go in pkg/kv.(*Txn).NegotiateAndSend at line 1519
pkg/kv/txn.go in pkg/kv.(*Txn).SetFixedTimestamp at line 1734
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).SetFixedTimestamp at line 1279
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v26.2.3 |
| Go Version | go1.25.5 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.2.3 |
| Cockroach SHA | 90d3b6080727dc810224a99903cb2e88b81e91ae |
| # of CPUs | 2 |
| # of Goroutines | 782 |

Contributor guide

Open the contributing guide

Research direction

Start at pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1279 and trace SetFixedTimestamp through pkg/kv/txn.go and pkg/sql/row/kv_fetcher.go. Investigate the reported AS OF SYSTEM TIME query and the transaction state showing reads before the fixed timestamp was set. Done means the assertion no longer occurs for this execution path and a regression test covers the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.