cockroachdb / cockroachdb/cockroach
sql/obs: provide a fast-path live blocker graph that does not require full `cluster_locks` fanout
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
On large, production-scale clusters (hundreds of user tables, thousands of ranges), the only way to answer "which transaction is blocking which transaction right now?" is to query `crdb_internal.cluster_locks WHERE contended = true`. This query performs a per-range RPC fanout (`QueryLocksRequest`) across every user table span via the descriptor walk in `crdb_internal.go`. On clusters at scale, this execution can take 2+ minutes even when it returns several rows.
There are several consequences to this behavior:
1. The DB Console's Active Transactions page is unusable for live contention triage on large clusters.
It fires the `cluster_locks` query every 10 seconds via auto-refresh. With a 2+ minute response time, the "Time Spent Waiting" column is silently minutes stale by construction: the column can only update when the query completes, so its displayed values always reflect the state from when the previous successful request began, not the present. In the Self-Hosted DB Console, the `CachedDataReducer` skips dispatch while a request is in-flight and enforces a 30-second invalidation period after completion (`apiReducers.ts`), so only one `cluster_locks` request runs at a time per tab. However, neither code path wires an `AbortController` through `fetch()`, so if an operator navigates away from the page mid-query, the server-side fanout runs to completion with no consumer for the result. The core issue is not request pileup but the query's inherent cost: even a single in-flight request takes 2+ minutes on a large cluster, during which the column displays stale or no data, and the server is doing a full per-range RPC traversal that cannot be cancelled from the client.
2. There is no alternative fast path.
The contention event tables (`cluster_contention_events`, `transaction_contention_events`) are post-hoc bounded caches of completed contention events (Least Recently Used, or LRU, for the former, FIFO for the latter) populated when a waiter unblocks. They cannot answer _"who is blocking right now."_ The data needed for a live blocker graph (waiting readers, queued locking requests) exists in-memory in each replica's `keyLocks` structure, where the only way to read it from SQL is via the `cluster_locks` vtable, which fans out across all ranges for every table the user has access to unless narrowed by a `table_id`, `table_name`, `database_name`, or `contended` filter. The underlying `QueryLocksRequest` KV command can also be issued programmatically against arbitrary key spans, but that path is not exposed to SQL users.
3. Engineering has previously acknowledged that the fanout is expensive. `cluster_locks` was removed from cockroach Debug Zip in [#114088 pkg/cli: Improve the latency impact of debug zip](https://github.com/cockroachdb/cockroach/pull/114088), with [#114215 debug: add flag to allow user to add cluster_locks table to debug zip #114215](https://github.com/cockroachdb/cockroach/issues/114215) tracking an opt-in flag to re-include it. Despite the removal in the Debug Zip, the DB Console's Active Transactions page continues to auto-refresh against the same query.
**Describe the solution you'd like**
A fast-path mechanism to retrieve the live blocker graph (which transactions are waiting on which transactions, on which keys) without requiring a full-cluster per-range fanout. Possible approaches:
1. Wire an `AbortController` through `fetchDataJSON` so that navigating away from the page cancels the in-flight server-side work. `fetchData.ts` currently issues `fetch(url, params)` with no signal on all branches through master, so a 2+ minute fanout runs to completion on the server even after the operator has moved on. Additionally, _surface the staleness in the UI_: show an explicit loading state with last-successful timestamp for the "Time Spent Waiting" column, rather than displaying silently stale data.
2. A purpose-built status endpoint (e.g., `/_status/blockers` or a new vtable) that enumerates only ranges with non-empty lock wait queues. The data already lives in each replica's concurrency manager (`keyLocks.waitingReaders`, `queuedLockingRequests`). A fast path could have each node's lock table maintain an index of ranges with active waiters, then resolve key-to-table mappings only for those hits, avoiding the descriptor walk entirely. This would make the cost O(contended ranges) instead of O(all ranges). Also consider that the reverse mapping from contended keys to table metadata is mechanically straightforward via `keys.DecodeTenantPrefix` and `keys.DecodeTableIDIndexID`, so the resolution cost scales with the number of contended keys, not the size of the catalog.
**Describe alternatives you've considered**
At present, the operator could consider filtering `cluster_locks` by `table_id`, `table_name`, `database_name`, and/or `contended`. This can help reduce the fanout scope to a single table's ranges, making this a relatively viable workaround via the SQL CLI. It is, however, a manual operation, so it requires the operator to already know which table is contended (and doesn't actively address the DB Console view).
```sql
SELECT
lock_key_pretty,
txn_id,
granted,
duration
FROM crdb_internal.cluster_locks
WHERE table_name = 'orders'
AND database_name = 'myDB'
AND contended = true;
```
While not technically an improvement to the page, **new to v26.2** is the [Active Session History (ASH)](https://www.cockroachlabs.com/docs/v26.2/active-session-history) feature. ASH can quickly identify that lock contention is occurring and which statement fingerprints are affected (via `work_event_type = 'LOCK'`) since it reads from in-memory ring buffers rather than doing a per-range fanout. However, ASH does not record blocker identity (no blocking `txn_id`), the specific contended key, or the blocker-waiter graph. It answers "which workloads are waiting on locks?" but not "who is blocking whom?" It is a useful complement for initial triage but not a substitute for the live blocker graph this request is about. See [what else is new in v26.2](https://www.cockroachlabs.com/docs/releases/v26.2).
**Additional context**
* The cluster_locks vtable comment in `crdb_internal.go` warns: *"Querying this table is an expensive operation since it creates a cluster-wide RPC-fanout."*
* The SWR migration on master ([#166571](https://github.com/cockroachdb/cockroach/pull/166571) for Active Statements, [#166834](https://github.com/cockroachdb/cockroach/pull/166834) for Active Transactions; neither is in 26.1 or 26.2, targeting 26.3.0) coincidentally reduces per-tab request accumulation as an inherent property of SWR's in-flight deduplication, but does not address the underlying fanout latency or the missing AbortController.
* Original cluster_locks + Active Executions integration: [#85081](https://github.com/cockroachdb/cockroach/pull/85081) (v22.2).
* Related issues: [#114215](https://github.com/cockroachdb/cockroach/issues/114215) (open; add opt-in flag for cluster_locks in debug zip), [#114088](https://github.com/cockroachdb/cockroach/pull/114088) (debug zip latency improvements, which disabled cluster_locks collection).
* This request originates from a support ticket on a self-hosted enterprise deployment at scale. The operator workflow for *"who is blocking whom right now?"* has no viable fast path on large clusters today.
Jira issue: CRDB-63324
Epic CRDB-55226
Contributor guide
Assessment
This issue has not been assessed yet.