cockroachdb / cockroachdb/cockroach
Sentry: txn_lock_gatekeeper.go:65: get-system-privileges: concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/6756798120/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6756798120/?referrer=webhooks_plugin)
Panic Message:
```
txn_lock_gatekeeper.go:65: get-system-privileges: concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760]
(1) secondary error attachment
| get-system-privileges: concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760]
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| | pkg/sql/internal.go:1244
| | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Close
| | pkg/sql/internal.go:605
| | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| | pkg/sql/internal.go:581
| | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
| | pkg/sql/syntheticprivilegecache/cache.go:144
| | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).Get.func1
| | pkg/sql/syntheticprivilegecache/cache.go:88
| | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
| | pkg/util/syncutil/singleflight/singleflight.go:384
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
| | pkg/util/stop/stopper.go:316
| | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
| | pkg/util/syncutil/singleflight/singleflight.go:383
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (2) get-system-privileges
| Wraps: (3)
| Wraps: (4) assertion failure
| Wraps: (5)
| | (opaque error wrapper)
| | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| | reportable 0:
| |
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:65
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
| | pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:234
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:144
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:334
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:112
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:265
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
| | pkg/kv/kvclient/kvcoord/txn_coord_sender.go:543
| | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
| | pkg/kv/db.go:1159
| | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
| | pkg/kv/txn.go:1302
| | github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.func2
| | pkg/sql/row/kv_batch_fetcher.go:285
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
| | pkg/sql/row/kv_batch_fetcher.go:647
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
| | pkg/sql/row/kv_batch_fetcher.go:940
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch
| | pkg/sql/row/kv_batch_fetcher.go:807
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).nextKV
| | pkg/sql/row/kv_fetcher.go:306
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
| | pkg/sql/row/kv_fetcher.go:327
| | github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch
| | pkg/sql/colfetcher/cfetcher.go:726
| | github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next
| | pkg/sql/colfetcher/colbatch_scan.go:238
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next
| | pkg/sql/colexec/colexecutils/cancel_checker.go:54
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| | pkg/sql/colexec/colexecbase/simple_project.go:113
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| | pkg/sql/colflow/stats.go:113
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | pkg/sql/colexecerror/error.go:162
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| | pkg/sql/colflow/stats.go:121
| | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next
| | pkg/sql/colexec/materializer.go:246
| | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter
| | pkg/sql/colexec/materializer.go:271
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | pkg/sql/colexecerror/error.go:162
| | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next
| | pkg/sql/colexec/materializer.go:277
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).next
| | pkg/sql/colflow/flow_coordinator.go:119
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).nextAdapter
| | pkg/sql/colflow/flow_coordinator.go:136
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | pkg/sql/colexecerror/error.go:162
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Next
| | pkg/sql/colflow/flow_coordinator.go:141
| Wraps: (6) concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760]
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *colexecerror.StorageError (4) *assert.withAssertionFailure (5) *errbase.opaqueWrapper (6) *errutil.leafError
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| pkg/sql/internal.go:1244
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Close
| pkg/sql/internal.go:605
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| pkg/sql/internal.go:581
| github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
| pkg/sql/syntheticprivilegecache/cache.go:144
| github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).Get.func1
| pkg/sql/syntheticprivilegecache/cache.go:88
| github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
| pkg/util/syncutil/singleflight/singleflight.go:384
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
| pkg/util/stop/stopper.go:316
| github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
| pkg/util/syncutil/singleflight/singleflight.go:383
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) get-system-privileges
Wraps: (4)
Wraps: (5) assertion failure
Wraps: (6)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
| pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:65
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
...
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/syncutil/singleflight/singleflight.go#L382-L384](pkg/util/syncutil/singleflight/singleflight.go#L382-L384)
[pkg/util/stop/stopper.go#L315-L317](pkg/util/stop/stopper.go#L315-L317)
[pkg/util/syncutil/singleflight/singleflight.go#L383-L385](pkg/util/syncutil/singleflight/singleflight.go#L383-L385)
[pkg/sql/syntheticprivilegecache/cache.go#L87-L89](pkg/sql/syntheticprivilegecache/cache.go#L87-L89)
[pkg/sql/syntheticprivilegecache/cache.go#L143-L145](pkg/sql/syntheticprivilegecache/cache.go#L143-L145)
[pkg/sql/internal.go#L580-L582](pkg/sql/internal.go#L580-L582)
[pkg/sql/internal.go#L604-L606](pkg/sql/internal.go#L604-L606)
[pkg/sql/internal.go#L1243-L1245](pkg/sql/internal.go#L1243-L1245)
[pkg/sql/colflow/flow_coordinator.go#L140-L142](pkg/sql/colflow/flow_coordinator.go#L140-L142)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/flow_coordinator.go#L135-L137](pkg/sql/colflow/flow_coordinator.go#L135-L137)
[pkg/sql/colflow/flow_coordinator.go#L118-L120](pkg/sql/colflow/flow_coordinator.go#L118-L120)
[pkg/sql/colexec/materializer.go#L276-L278](pkg/sql/colexec/materializer.go#L276-L278)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colexec/materializer.go#L270-L272](pkg/sql/colexec/materializer.go#L270-L272)
[pkg/sql/colexec/materializer.go#L245-L247](pkg/sql/colexec/materializer.go#L245-L247)
[pkg/sql/colflow/stats.go#L120-L122](pkg/sql/colflow/stats.go#L120-L122)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L112-L114](pkg/sql/colflow/stats.go#L112-L114)
[pkg/sql/colexec/colexecbase/simple_project.go#L112-L114](pkg/sql/colexec/colexecbase/simple_project.go#L112-L114)
[pkg/sql/colexec/colexecutils/cancel_checker.go#L53-L55](pkg/sql/colexec/colexecutils/cancel_checker.go#L53-L55)
[pkg/sql/colfetcher/colbatch_scan.go#L237-L239](pkg/sql/colfetcher/colbatch_scan.go#L237-L239)
[pkg/sql/colfetcher/cfetcher.go#L725-L727](pkg/sql/colfetcher/cfetcher.go#L725-L727)
[pkg/sql/row/kv_fetcher.go#L326-L328](pkg/sql/row/kv_fetcher.go#L326-L328)
[pkg/sql/row/kv_fetcher.go#L305-L307](pkg/sql/row/kv_fetcher.go#L305-L307)
[pkg/sql/row/kv_batch_fetcher.go#L806-L808](pkg/sql/row/kv_batch_fetcher.go#L806-L808)
[pkg/sql/row/kv_batch_fetcher.go#L939-L941](pkg/sql/row/kv_batch_fetcher.go#L939-L941)
[pkg/sql/row/kv_batch_fetcher.go#L646-L648](pkg/sql/row/kv_batch_fetcher.go#L646-L648)
[pkg/sql/row/kv_batch_fetcher.go#L284-L286](pkg/sql/row/kv_batch_fetcher.go#L284-L286)
[pkg/kv/txn.go#L1301-L1303](pkg/kv/txn.go#L1301-L1303)
[pkg/kv/db.go#L1158-L1160](pkg/kv/db.go#L1158-L1160)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L542-L544](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L542-L544)
[pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L264-L266](pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L264-L266)
[pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L111-L113](pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L111-L113)
[pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L333-L335](pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L333-L335)
[pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L143-L145](pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L143-L145)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L161-L163](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L161-L163)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L233-L235](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L233-L235)
[pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go#L40-L42](pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go#L40-L42)
[pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go#L64-L66](pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go#L64-L66)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall at line 383
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTask at line 316
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall.func1 at line 384
pkg/sql/syntheticprivilegecache/cache.go in pkg/sql/syntheticprivilegecache.(*Cache).Get.func1 at line 88
pkg/sql/syntheticprivilegecache/cache.go in pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage at line 144
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Next at line 581
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Close at line 605
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).execInternal.func1.1 at line 1244
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Next at line 141
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).nextAdapter at line 136
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).next at line 119
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Next at line 277
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).nextAdapter at line 271
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).next at line 246
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 121
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 113
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 113
pkg/sql/colexec/colexecutils/cancel_checker.go in pkg/sql/colexec/colexecutils.(*CancelChecker).Next at line 54
pkg/sql/colfetcher/colbatch_scan.go in pkg/sql/colfetcher.(*ColBatchScan).Next at line 238
pkg/sql/colfetcher/cfetcher.go in pkg/sql/colfetcher.(*cFetcher).NextBatch at line 726
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).NextKV at line 327
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).nextKV at line 306
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).NextBatch at line 807
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).nextBatch at line 940
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).fetch at line 647
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.newTxnKVFetcher.makeSendFunc.func2 at line 285
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1302
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1159
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 543
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go in pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked at line 265
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked at line 112
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked at line 334
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go in pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked at line 144
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked at line 162
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts at line 234
pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go in pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked at line 41
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go in pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked at line 65
```
### Tags
| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.1.1 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.1.1 |
| Cockroach SHA | a5c682ed7f4b1e42717616d676ff14d6e6a8b192 |
| # of CPUs | 12 |
| # of Goroutines | 517 |
Jira issue: CRDB-52763
Contributor guide
Research direction
Start at pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:65 and follow the reported path through pkg/sql/syntheticprivilegecache/cache.go:144, pkg/sql/internal.go:1244, and pkg/kv/txn.go:1302. Reproduce or isolate the concurrent transaction use reported by Sentry, then verify the resulting fix with regression coverage showing that the panic no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100