cockroachdb / cockroachdb/cockroach
sql/inspect: per-job protected timestamp Protect() doesn't scale to many concurrent INSPECT jobs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
When many INSPECT jobs run concurrently on a large cluster, each job's per-job `Protect()` call to the protected timestamps subsystem contends on the single `system.protected_ts_meta` version row. At sufficient concurrency, `Protect()` calls time out and the cluster degrades severely.
**To Reproduce**
On a cluster with many tables, launch many concurrent INSPECT jobs. Observed on the shared `proj-1m-tables-100k` cluster (~100K tables, ~14K concurrent INSPECT jobs) on 2026-05-15. Reproduction in a controlled setting requires high concurrent-INSPECT-job count.
**Expected behavior**
INSPECT jobs should complete (or fail cleanly) without timing out on protected timestamp creation, and shouldn't cause cluster-wide degradation. The system should scale to support concurrent INSPECT across many databases — the supported use case is many DBs × normal tables each, where a customer might run scheduled INSPECT across all their DBs.
**Additional data / context**
Symptom (see [#proj-1m-tables Slack thread, 2026-05-15](https://cockroachlabs.enterprise.slack.com/archives/C0APP6FTMPF/p1778858061673389)):
```
failed to protect timestamp 1778810797.351350543,0 for INSPECT job 1175520534222405637: job 1175520534222405637: failed to write record 86300859-b2c9-482f-ad95-17b95e1aa061: protectedts-protect: operation "intent_resolver_ir_batcher.sendBatch" timed out after 1m0.001s (given timeout 1m0s): aborted in DistSender: result is ambiguous: context deadline exceeded
```
Per @stevendanna's live investigation:
- n2 (leaseholder of `system.protected_ts_meta`) at 94%+ CPU.
- ~20K goroutines waiting on latches against the single table.
- ~16K queued transactions with application name `$ internal-protectedts-protect`.
- `protected_ts_meta.version` advanced only 1160 times despite thousands of concurrent attempts — the unlocked-read-then-upsert pattern causes WriteTooOld churn where most contending transactions never make forward progress.
- Leaseholder for the table thrashed under the load, exacerbating contention.
**Mechanics:**
- [`pkg/sql/inspect/inspect_resumer.go:60`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/inspect/inspect_resumer.go#L60) calls `maybeProtectTimestamp` at resume.
- [`pkg/sql/inspect/inspect_resumer.go:445-461`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/inspect/inspect_resumer.go#L445-L461) calls `execCfg.ProtectedTimestampManager.Protect(ctx, c.job, target, details.AsOf)` once per job.
Every concurrent INSPECT job races independently against `protected_ts_meta`. Other job types take per-job PTS records too (backup, schemachange, IMPORT) but rarely have thousands of jobs concurrently.
**Design questions for triage:**
- Can multiple concurrent INSPECT jobs covering the same/overlapping targets share a PTS record?
- Should there be a cluster-wide cap on concurrent INSPECT jobs?
- Can `Protect()` calls be batched, or backoff/retry on contention?
The intrinsic `protected_ts_meta` single-row-upsert contention pattern may itself deserve attention from KV / protected-timestamps owners, but this issue is scoped to the INSPECT-side fix.
**Environment:**
- CockroachDB version: `v26.3.0-alpha.00000000-dev-542a5b01648a69653f10e32a9f510b3dd008c01d`
- Cluster: shared `proj-1m-tables-100k`, ~100K tables, 9 nodes (n2-highmem-16).
**Additional context**
@stevendanna captured a trace and many logs from the 2026-05-15 incident — please coordinate with him before triage.
Related: CRDB-63472 (closed as out-of-scope; this issue captures what was actually surfaced when INSPECT was run on the 100K cluster), CRDB-64042.
Epic CRDB-58778
Jira issue: CRDB-64578
Contributor guide
Research direction
Start in pkg/sql/inspect/inspect_resumer.go at maybeProtectTimestamp and lines 445-461, then trace the ProtectedTimestampManager.Protect call and the protected_ts_meta contention described in the incident data. Coordinate with @stevendanna before triage. Done means concurrent INSPECT jobs complete or fail cleanly without protected-timestamp timeouts or cluster-wide degradation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100