cockroachdb / cockroachdb/cockroach
Sentry: txn_coord_sender.go:732: LeafTxn "sql txn" meta={id=23254a89 key=/Min iso=Serializable pri=0.00333516 epo=0 ts=1789654284.200978312,0 min=1789654284.200978312,0 seq=0} lock=false stat=PENDING ...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7738039697/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7738039697/?referrer=webhooks_plugin)
Panic Message:
```
txn_coord_sender.go:732: LeafTxn "sql txn" meta={id=23254a89 key=/Min iso=Serializable pri=0.00333516 epo=0 ts=1789654284.200978312,0 min=1789654284.200978312,0 seq=0} lock=false stat=PENDING rts=1789654284.200978312,0 gul=1789654284.700978312,0 obs={n1@1789654284.200978312,0} incompatible with locking request ×
(1) plan gist: AgGyLRoAAwAAAhOyLQIHAgYC
Wraps: (2) while executing: SELECT _._(_._, gen_random_uuid(), _._, _, _) FROM _ AS _ WHERE _._ IS NULL LIMIT _
Wraps: (3)
Wraps: (4)
Wraps: (5)
Wraps: (6) tags: [n×,client=×,hostssl=,user=×]
Wraps: (7) assertion failure
Wraps: (8)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).maybeRejectIncompatibleRequest
| pkg/kv/kvclient/kvcoord/txn_coord_sender.go:732
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
| pkg/kv/kvclient/kvcoord/txn_coord_sender.go:522
| github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
| pkg/kv/db.go:1246
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
| pkg/kv/txn.go:1419
| github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.func2
| pkg/sql/row/kv_batch_fetcher.go:292
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
| pkg/sql/row/kv_batch_fetcher.go:674
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
| pkg/sql/row/kv_batch_fetcher.go:964
| github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch
| pkg/sql/row/kv_batch_fetcher.go:831
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).nextKV
| pkg/sql/row/kv_fetcher.go:334
| github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
| pkg/sql/row/kv_fetcher.go:355
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch
| pkg/sql/colfetcher/cfetcher.go:785
| github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next
| pkg/sql/colfetcher/colbatch_scan.go:270
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next
| pkg/sql/colexec/colexecutils/cancel_checker.go:62
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*isNullSelOp).Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go:220
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecsel.(*selEQBytesBytesConstOp).Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go:360
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*metadataHelper).next
| pkg/sql/colexec/colexecjoin/joiner_util.go:376
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*metadataHelper).inputTwoNext
| pkg/sql/colexec/colexecjoin/joiner_util.go:390
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*crossJoiner).consumeRightInput
| pkg/sql/colexec/colexecjoin/crossjoiner.go:194
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin.(*crossJoiner).Next
| pkg/sql/colexec/colexecjoin/crossjoiner.go:102
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| pkg/sql/colexec/colexecbase/simple_project.go:114
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| pkg/sql/colflow/stats.go:122
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| pkg/sql/colexecerror/error.go:162
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| pkg/sql/colflow/stats.go:130
| github.com/cockroachdb/cockroach/pkg/sql/colexecop.(*noopOperator).Next
| pkg/sql/colexecop/operator.go:476
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
| pkg/sql/colexec/colexecutils/operator.go:153
| github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*castStringStringOp).Next
| bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go:11381
Wraps: (9) LeafTxn "sql txn" meta={id=23254a89 key=/Min iso=Serializable pri=0.00333516 epo=0 ts=1789654284.200978312,0 min=1789654284.200978312,0 seq=0} lock=false stat=PENDING rts=1789654284.200978312,0 gul=1789654284.700978312,0 obs={n1@1789654284.200978312,0} incompatible with locking request ×
Error types: (1) *safedetails.withSafeDetails (2) *safedetails.withSafeDetails (3) *colexecerror.notInternalError (4) *colexecerror.notInternalError (5) *colexecerror.StorageError (6) *contexttags.withContext (7) *assert.withAssertionFailure (8) *errbase.opaqueWrapper (9) *errutil.leafError
-- report composition:
*errutil.leafError: LeafTxn "sql txn" meta={id=23254a89 key=/Min iso=Serializable pri=0.00333516 epo=0 ts=1789654284.200978312,0 min=1789654284.200978312,0 seq=0} lock=false stat=PENDING rts=1789654284.200978312,0 gul=1789654284.700978312,0 obs={n1@1789654284.200978312,0} incompatible with locking request ×
txn_coord_sender.go:732: *withstack.withStack (top exception)
*assert.withAssertionFailure
*contexttags.withContext: n1
*colexecerror.StorageError
*colexecerror.notInternalError
*colexecerror.notInternalError
*safedetails.withSafeDetails: while executing: SELECT _._(_._, gen_random_uuid(), _._, _, _) FROM _ AS _ WHERE _._ IS NULL LIMIT _
*safedetails.withSafeDetails: plan gist: AgGyLRoAAwAAAhOyLQIHAgYC
```
Stacktrace (expand for inline code snippets):
[bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L11380-L11382](bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go#L11380-L11382)
[pkg/sql/colexec/colexecutils/operator.go#L152-L154](pkg/sql/colexec/colexecutils/operator.go#L152-L154)
[pkg/sql/colexecop/operator.go#L475-L477](pkg/sql/colexecop/operator.go#L475-L477)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/colexecbase/simple_project.go#L113-L115](pkg/sql/colexec/colexecbase/simple_project.go#L113-L115)
[pkg/sql/colexec/colexecjoin/crossjoiner.go#L101-L103](pkg/sql/colexec/colexecjoin/crossjoiner.go#L101-L103)
[pkg/sql/colexec/colexecjoin/crossjoiner.go#L193-L195](pkg/sql/colexec/colexecjoin/crossjoiner.go#L193-L195)
[pkg/sql/colexec/colexecjoin/joiner_util.go#L389-L391](pkg/sql/colexec/colexecjoin/joiner_util.go#L389-L391)
[pkg/sql/colexec/colexecjoin/joiner_util.go#L375-L377](pkg/sql/colexec/colexecjoin/joiner_util.go#L375-L377)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go#L359-L361](bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go#L359-L361)
[bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go#L219-L221](bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go#L219-L221)
[pkg/sql/colflow/stats.go#L129-L131](pkg/sql/colflow/stats.go#L129-L131)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L121-L123](pkg/sql/colflow/stats.go#L121-L123)
[pkg/sql/colexec/colexecutils/cancel_checker.go#L61-L63](pkg/sql/colexec/colexecutils/cancel_checker.go#L61-L63)
[pkg/sql/colfetcher/colbatch_scan.go#L269-L271](pkg/sql/colfetcher/colbatch_scan.go#L269-L271)
[pkg/sql/colfetcher/cfetcher.go#L784-L786](pkg/sql/colfetcher/cfetcher.go#L784-L786)
[pkg/sql/row/kv_fetcher.go#L354-L356](pkg/sql/row/kv_fetcher.go#L354-L356)
[pkg/sql/row/kv_fetcher.go#L333-L335](pkg/sql/row/kv_fetcher.go#L333-L335)
[pkg/sql/row/kv_batch_fetcher.go#L830-L832](pkg/sql/row/kv_batch_fetcher.go#L830-L832)
[pkg/sql/row/kv_batch_fetcher.go#L963-L965](pkg/sql/row/kv_batch_fetcher.go#L963-L965)
[pkg/sql/row/kv_batch_fetcher.go#L673-L675](pkg/sql/row/kv_batch_fetcher.go#L673-L675)
[pkg/sql/row/kv_batch_fetcher.go#L291-L293](pkg/sql/row/kv_batch_fetcher.go#L291-L293)
[pkg/kv/txn.go#L1418-L1420](pkg/kv/txn.go#L1418-L1420)
[pkg/kv/db.go#L1245-L1247](pkg/kv/db.go#L1245-L1247)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L521-L523](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L521-L523)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L731-L733](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L731-L733)
```
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/cast.eg.go in pkg/sql/colexec/colexecbase.(*castStringStringOp).Next at line 11381
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 153
pkg/sql/colexecop/operator.go in pkg/sql/colexecop.(*noopOperator).Next at line 476
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 114
pkg/sql/colexec/colexecjoin/crossjoiner.go in pkg/sql/colexec/colexecjoin.(*crossJoiner).Next at line 102
pkg/sql/colexec/colexecjoin/crossjoiner.go in pkg/sql/colexec/colexecjoin.(*crossJoiner).consumeRightInput at line 194
pkg/sql/colexec/colexecjoin/joiner_util.go in pkg/sql/colexec/colexecjoin.(*metadataHelper).inputTwoNext at line 390
pkg/sql/colexec/colexecjoin/joiner_util.go in pkg/sql/colexec/colexecjoin.(*metadataHelper).next at line 376
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecsel/selection_ops.eg.go in pkg/sql/colexec/colexecsel.(*selEQBytesBytesConstOp).Next at line 360
bazel-out/k8-opt/bin/pkg/sql/colexec/is_null_ops.eg.go in pkg/sql/colexec.(*isNullSelOp).Next at line 220
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 130
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 122
pkg/sql/colexec/colexecutils/cancel_checker.go in pkg/sql/colexec/colexecutils.(*CancelChecker).Next at line 62
pkg/sql/colfetcher/colbatch_scan.go in pkg/sql/colfetcher.(*ColBatchScan).Next at line 270
pkg/sql/colfetcher/cfetcher.go in pkg/sql/colfetcher.(*cFetcher).NextBatch at line 785
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).NextKV at line 355
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).nextKV at line 334
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).NextBatch at line 831
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).nextBatch at line 964
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).fetch at line 674
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.newTxnKVFetcher.makeSendFunc.func2 at line 292
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1419
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1246
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 522
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).maybeRejectIncompatibleRequest at line 732
```
### Tags
| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v26.2.1 |
| Go Version | go1.25.5 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.2.1 |
| Cockroach SHA | b398a6e1fde74a915daf8e9841303c8ee39c2abe |
| # of CPUs | 24 |
| # of Goroutines | 693 |
Jira issue: CRDB-68474
Contributor guide
Research direction
Read pkg/kv/kvclient/kvcoord/txn_coord_sender.go around maybeRejectIncompatibleRequest and Send, then trace the SQL path through pkg/sql/row/kv_batch_fetcher.go. Use the Sentry panic and reported SELECT as the starting case; done requires identifying the cause of the incompatible locking request and adding a regression test, but the issue does not name a reproducer or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100