cockroachdb / cockroachdb/cockroach

Sentry: txn_lock_gatekeeper.go:65: get-system-privileges: concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760...

Open
#150,552 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.1 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/6756798120/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/6756798120/?referrer=webhooks_plugin)

Panic Message:

```
txn_lock_gatekeeper.go:65: get-system-privileges: concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760]
(1) secondary error attachment
| get-system-privileges: concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760]
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| | pkg/sql/internal.go:1244
| | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Close
| | pkg/sql/internal.go:605
| | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| | pkg/sql/internal.go:581
| | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
| | pkg/sql/syntheticprivilegecache/cache.go:144
| | github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).Get.func1
| | pkg/sql/syntheticprivilegecache/cache.go:88
| | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
| | pkg/util/syncutil/singleflight/singleflight.go:384
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
| | pkg/util/stop/stopper.go:316
| | github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
| | pkg/util/syncutil/singleflight/singleflight.go:383
| | runtime.goexit
| | src/runtime/asm_amd64.s:1695
| Wraps: (2) get-system-privileges
| Wraps: (3)
| Wraps: (4) assertion failure
| Wraps: (5)
| | (opaque error wrapper)
| | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| | reportable 0:
| |
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:65
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
| | pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:234
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:144
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:334
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:112
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked
| | pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:265
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send
| | pkg/kv/kvclient/kvcoord/txn_coord_sender.go:543
| | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
| | pkg/kv/db.go:1159
| | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send
| | pkg/kv/txn.go:1302
| | github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeSendFunc.func2
| | pkg/sql/row/kv_batch_fetcher.go:285
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
| | pkg/sql/row/kv_batch_fetcher.go:647
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
| | pkg/sql/row/kv_batch_fetcher.go:940
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch
| | pkg/sql/row/kv_batch_fetcher.go:807
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).nextKV
| | pkg/sql/row/kv_fetcher.go:306
| | github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
| | pkg/sql/row/kv_fetcher.go:327
| | github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*cFetcher).NextBatch
| | pkg/sql/colfetcher/cfetcher.go:726
| | github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchScan).Next
| | pkg/sql/colfetcher/colbatch_scan.go:238
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next
| | pkg/sql/colexec/colexecutils/cancel_checker.go:54
| | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
| | pkg/sql/colexec/colexecbase/simple_project.go:113
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
| | pkg/sql/colflow/stats.go:113
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | pkg/sql/colexecerror/error.go:162
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
| | pkg/sql/colflow/stats.go:121
| | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next
| | pkg/sql/colexec/materializer.go:246
| | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter
| | pkg/sql/colexec/materializer.go:271
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | pkg/sql/colexecerror/error.go:162
| | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next
| | pkg/sql/colexec/materializer.go:277
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).next
| | pkg/sql/colflow/flow_coordinator.go:119
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).nextAdapter
| | pkg/sql/colflow/flow_coordinator.go:136
| | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| | pkg/sql/colexecerror/error.go:162
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Next
| | pkg/sql/colflow/flow_coordinator.go:141
| Wraps: (6) concurrent txn use detected. ba: Scan [/Table/52/2/×,/Table/52/2/×/×), [txn: daedc42e], [max_span_request_keys: 0], [target_bytes: 10485760]
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *colexecerror.StorageError (4) *assert.withAssertionFailure (5) *errbase.opaqueWrapper (6) *errutil.leafError
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| pkg/sql/internal.go:1244
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Close
| pkg/sql/internal.go:605
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| pkg/sql/internal.go:581
| github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage
| pkg/sql/syntheticprivilegecache/cache.go:144
| github.com/cockroachdb/cockroach/pkg/sql/syntheticprivilegecache.(*Cache).Get.func1
| pkg/sql/syntheticprivilegecache/cache.go:88
| github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall.func1
| pkg/util/syncutil/singleflight/singleflight.go:384
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask
| pkg/util/stop/stopper.go:316
| github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall
| pkg/util/syncutil/singleflight/singleflight.go:383
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (3) get-system-privileges
Wraps: (4)
Wraps: (5) assertion failure
Wraps: (6)
| (opaque error wrapper)
| type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| reportable 0:
|
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked
| pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:65
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked
| pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts
...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/syncutil/singleflight/singleflight.go#L382-L384](pkg/util/syncutil/singleflight/singleflight.go#L382-L384)
[pkg/util/stop/stopper.go#L315-L317](pkg/util/stop/stopper.go#L315-L317)
[pkg/util/syncutil/singleflight/singleflight.go#L383-L385](pkg/util/syncutil/singleflight/singleflight.go#L383-L385)
[pkg/sql/syntheticprivilegecache/cache.go#L87-L89](pkg/sql/syntheticprivilegecache/cache.go#L87-L89)
[pkg/sql/syntheticprivilegecache/cache.go#L143-L145](pkg/sql/syntheticprivilegecache/cache.go#L143-L145)
[pkg/sql/internal.go#L580-L582](pkg/sql/internal.go#L580-L582)
[pkg/sql/internal.go#L604-L606](pkg/sql/internal.go#L604-L606)
[pkg/sql/internal.go#L1243-L1245](pkg/sql/internal.go#L1243-L1245)
[pkg/sql/colflow/flow_coordinator.go#L140-L142](pkg/sql/colflow/flow_coordinator.go#L140-L142)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/flow_coordinator.go#L135-L137](pkg/sql/colflow/flow_coordinator.go#L135-L137)
[pkg/sql/colflow/flow_coordinator.go#L118-L120](pkg/sql/colflow/flow_coordinator.go#L118-L120)
[pkg/sql/colexec/materializer.go#L276-L278](pkg/sql/colexec/materializer.go#L276-L278)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colexec/materializer.go#L270-L272](pkg/sql/colexec/materializer.go#L270-L272)
[pkg/sql/colexec/materializer.go#L245-L247](pkg/sql/colexec/materializer.go#L245-L247)
[pkg/sql/colflow/stats.go#L120-L122](pkg/sql/colflow/stats.go#L120-L122)
[pkg/sql/colexecerror/error.go#L161-L163](pkg/sql/colexecerror/error.go#L161-L163)
[pkg/sql/colflow/stats.go#L112-L114](pkg/sql/colflow/stats.go#L112-L114)
[pkg/sql/colexec/colexecbase/simple_project.go#L112-L114](pkg/sql/colexec/colexecbase/simple_project.go#L112-L114)
[pkg/sql/colexec/colexecutils/cancel_checker.go#L53-L55](pkg/sql/colexec/colexecutils/cancel_checker.go#L53-L55)
[pkg/sql/colfetcher/colbatch_scan.go#L237-L239](pkg/sql/colfetcher/colbatch_scan.go#L237-L239)
[pkg/sql/colfetcher/cfetcher.go#L725-L727](pkg/sql/colfetcher/cfetcher.go#L725-L727)
[pkg/sql/row/kv_fetcher.go#L326-L328](pkg/sql/row/kv_fetcher.go#L326-L328)
[pkg/sql/row/kv_fetcher.go#L305-L307](pkg/sql/row/kv_fetcher.go#L305-L307)
[pkg/sql/row/kv_batch_fetcher.go#L806-L808](pkg/sql/row/kv_batch_fetcher.go#L806-L808)
[pkg/sql/row/kv_batch_fetcher.go#L939-L941](pkg/sql/row/kv_batch_fetcher.go#L939-L941)
[pkg/sql/row/kv_batch_fetcher.go#L646-L648](pkg/sql/row/kv_batch_fetcher.go#L646-L648)
[pkg/sql/row/kv_batch_fetcher.go#L284-L286](pkg/sql/row/kv_batch_fetcher.go#L284-L286)
[pkg/kv/txn.go#L1301-L1303](pkg/kv/txn.go#L1301-L1303)
[pkg/kv/db.go#L1158-L1160](pkg/kv/db.go#L1158-L1160)
[pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L542-L544](pkg/kv/kvclient/kvcoord/txn_coord_sender.go#L542-L544)
[pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L264-L266](pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go#L264-L266)
[pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L111-L113](pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go#L111-L113)
[pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L333-L335](pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go#L333-L335)
[pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L143-L145](pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go#L143-L145)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L161-L163](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L161-L163)
[pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L233-L235](pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go#L233-L235)
[pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go#L40-L42](pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go#L40-L42)
[pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go#L64-L66](pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go#L64-L66)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall at line 383
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTask at line 316
pkg/util/syncutil/singleflight/singleflight.go in pkg/util/syncutil/singleflight.(*Group).doCall.func1 at line 384
pkg/sql/syntheticprivilegecache/cache.go in pkg/sql/syntheticprivilegecache.(*Cache).Get.func1 at line 88
pkg/sql/syntheticprivilegecache/cache.go in pkg/sql/syntheticprivilegecache.(*Cache).readFromStorage at line 144
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Next at line 581
pkg/sql/internal.go in pkg/sql.(*rowsIterator).Close at line 605
pkg/sql/internal.go in pkg/sql.(*InternalExecutor).execInternal.func1.1 at line 1244
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).Next at line 141
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).nextAdapter at line 136
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*FlowCoordinator).next at line 119
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).Next at line 277
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).nextAdapter at line 271
pkg/sql/colexec/materializer.go in pkg/sql/colexec.(*Materializer).next at line 246
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 121
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 162
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 113
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 113
pkg/sql/colexec/colexecutils/cancel_checker.go in pkg/sql/colexec/colexecutils.(*CancelChecker).Next at line 54
pkg/sql/colfetcher/colbatch_scan.go in pkg/sql/colfetcher.(*ColBatchScan).Next at line 238
pkg/sql/colfetcher/cfetcher.go in pkg/sql/colfetcher.(*cFetcher).NextBatch at line 726
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).NextKV at line 327
pkg/sql/row/kv_fetcher.go in pkg/sql/row.(*KVFetcher).nextKV at line 306
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).NextBatch at line 807
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).nextBatch at line 940
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.(*txnKVFetcher).fetch at line 647
pkg/sql/row/kv_batch_fetcher.go in pkg/sql/row.newTxnKVFetcher.makeSendFunc.func2 at line 285
pkg/kv/txn.go in pkg/kv.(*Txn).Send at line 1302
pkg/kv/db.go in pkg/kv.(*DB).sendUsingSender at line 1159
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send at line 543
pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go in pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked at line 265
pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go in pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked at line 112
pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go in pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked at line 334
pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go in pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked at line 144
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked at line 162
pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts at line 234
pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go in pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked at line 41
pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go in pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked at line 65
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.1.1 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.1.1 |
| Cockroach SHA | a5c682ed7f4b1e42717616d676ff14d6e6a8b192 |
| # of CPUs | 12 |
| # of Goroutines | 517 |

Jira issue: CRDB-52763

Contributor guide

Open the contributing guide

Research direction

Start at pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:65 and follow the reported path through pkg/sql/syntheticprivilegecache/cache.go:144, pkg/sql/internal.go:1244, and pkg/kv/txn.go:1302. Reproduce or isolate the concurrent transaction use reported by Sentry, then verify the resulting fix with regression coverage showing that the panic no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.