cockroachdb / cockroachdb/cockroach

sql/inspect: per-job protected timestamp Protect() doesn't scale to many concurrent INSPECT jobs

Open
#171,467 1 comment 0 reactions 0 assignees View on GitHub
A-many-descriptors branch-master C-bug O-agent P-3 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

When many INSPECT jobs run concurrently on a large cluster, each job's per-job `Protect()` call to the protected timestamps subsystem contends on the single `system.protected_ts_meta` version row. At sufficient concurrency, `Protect()` calls time out and the cluster degrades severely.

**To Reproduce**

On a cluster with many tables, launch many concurrent INSPECT jobs. Observed on the shared `proj-1m-tables-100k` cluster (~100K tables, ~14K concurrent INSPECT jobs) on 2026-05-15. Reproduction in a controlled setting requires high concurrent-INSPECT-job count.

**Expected behavior**

INSPECT jobs should complete (or fail cleanly) without timing out on protected timestamp creation, and shouldn't cause cluster-wide degradation. The system should scale to support concurrent INSPECT across many databases — the supported use case is many DBs × normal tables each, where a customer might run scheduled INSPECT across all their DBs.

**Additional data / context**

Symptom (see [#proj-1m-tables Slack thread, 2026-05-15](https://cockroachlabs.enterprise.slack.com/archives/C0APP6FTMPF/p1778858061673389)):
```
failed to protect timestamp 1778810797.351350543,0 for INSPECT job 1175520534222405637: job 1175520534222405637: failed to write record 86300859-b2c9-482f-ad95-17b95e1aa061: protectedts-protect: operation "intent_resolver_ir_batcher.sendBatch" timed out after 1m0.001s (given timeout 1m0s): aborted in DistSender: result is ambiguous: context deadline exceeded
```

Per @stevendanna's live investigation:
- n2 (leaseholder of `system.protected_ts_meta`) at 94%+ CPU.
- ~20K goroutines waiting on latches against the single table.
- ~16K queued transactions with application name `$ internal-protectedts-protect`.
- `protected_ts_meta.version` advanced only 1160 times despite thousands of concurrent attempts — the unlocked-read-then-upsert pattern causes WriteTooOld churn where most contending transactions never make forward progress.
- Leaseholder for the table thrashed under the load, exacerbating contention.

**Mechanics:**
- [`pkg/sql/inspect/inspect_resumer.go:60`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/inspect/inspect_resumer.go#L60) calls `maybeProtectTimestamp` at resume.
- [`pkg/sql/inspect/inspect_resumer.go:445-461`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/inspect/inspect_resumer.go#L445-L461) calls `execCfg.ProtectedTimestampManager.Protect(ctx, c.job, target, details.AsOf)` once per job.

Every concurrent INSPECT job races independently against `protected_ts_meta`. Other job types take per-job PTS records too (backup, schemachange, IMPORT) but rarely have thousands of jobs concurrently.

**Design questions for triage:**
- Can multiple concurrent INSPECT jobs covering the same/overlapping targets share a PTS record?
- Should there be a cluster-wide cap on concurrent INSPECT jobs?
- Can `Protect()` calls be batched, or backoff/retry on contention?

The intrinsic `protected_ts_meta` single-row-upsert contention pattern may itself deserve attention from KV / protected-timestamps owners, but this issue is scoped to the INSPECT-side fix.

**Environment:**
- CockroachDB version: `v26.3.0-alpha.00000000-dev-542a5b01648a69653f10e32a9f510b3dd008c01d`
- Cluster: shared `proj-1m-tables-100k`, ~100K tables, 9 nodes (n2-highmem-16).

**Additional context**

@stevendanna captured a trace and many logs from the 2026-05-15 incident — please coordinate with him before triage.

Related: CRDB-63472 (closed as out-of-scope; this issue captures what was actually surfaced when INSPECT was run on the 100K cluster), CRDB-64042.

Epic CRDB-58778
Jira issue: CRDB-64578

Contributor guide

Open the contributing guide

Research direction

Start in pkg/sql/inspect/inspect_resumer.go at maybeProtectTimestamp and lines 445-461, then trace the ProtectedTimestampManager.Protect call and the protected_ts_meta contention described in the incident data. Coordinate with @stevendanna before triage. Done means concurrent INSPECT jobs complete or fail cleanly without protected-timestamp timeouts or cluster-wide degradation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.