cockroachdb / cockroachdb/cockroach

jobs: switch adoption from polling to rangefeed-based for scale

Open
#170,674 0 comments 0 reactions 1 assignee Assigned to @andrew-r-thomas View on GitHub
A-jobs C-enhancement P-1
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The jobs infrastructure uses polling-based adoption: every node runs a `SELECT ... FOR UPDATE` against `system.jobs` (`pkg/jobs/adopt.go:57-62`) every 30s (`defaultAdoptInterval`) and claims up to 10 jobs per loop (`maxAdoptionsPerLoop`). All nodes converge on the same `(status, created)` index, so at scale this generates severe contention on the hot rows and caps per-node adoption throughput at ~20 jobs/min, ~180 jobs/min cluster-wide on a 9-node cluster.

Concrete evidence the current design doesn't hold up at scale:

- Recent CDC testing with 100k changefeeds took **several hours** to cancel, batched in groups of 10k, due to lock contention on claim updates.
- Large-cluster deployments (80+ nodes) have hit the same class of contention.
- TTL and stats jobs at 1M tables would produce so much churn that operator visibility into the jobs system degrades: *"a million jobs every five minutes is an unpleasant situation to be in if you're trying to figure out what was going on in this cluster."*

### Proposed approach

Replace the polling claim query with a **rangefeed** on `system.jobs`. Each node's registry listens to the table. When a claimable job appears, the node uses a **hashed-job-ID shard heuristic** for back-off:

- Compute `hash(job_id) mod numNodes` against the current instance cache.
- Zero back-off if the job hashes to "your" shard.
- ~30s back-off otherwise.
- Whichever node claims first wins; other nodes observe the claim via the rangefeed and drop their attempt.

This eliminates the per-tick contention storm without needing a dedicated leasing layer.

### Migration shape

The new adopter should be built **purely additive** on top of the existing pollers, gated by a cluster setting:

- V1 keeps the existing pollers running as a safety net, with adaptive back-off — *"if the last 10 polls you didn't see anything to do, start polling once every 10 minutes."*
- If the rangefeed gets stuck, restarting it is acceptable degradation: worst case "five minutes to start instead of eight seconds," covered by the poller backstop.
- In tests, pollers can be turned off (or metamorphically toggled).
- Phased default-on: optional + metamorphic in 26.3, try in cloud, default-on in a later release.

Epic CRDB-62562

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.