cockroachdb / cockroachdb/cockroach
jobs: switch adoption from polling to rangefeed-based for scale
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The jobs infrastructure uses polling-based adoption: every node runs a `SELECT ... FOR UPDATE` against `system.jobs` (`pkg/jobs/adopt.go:57-62`) every 30s (`defaultAdoptInterval`) and claims up to 10 jobs per loop (`maxAdoptionsPerLoop`). All nodes converge on the same `(status, created)` index, so at scale this generates severe contention on the hot rows and caps per-node adoption throughput at ~20 jobs/min, ~180 jobs/min cluster-wide on a 9-node cluster.
Concrete evidence the current design doesn't hold up at scale:
- Recent CDC testing with 100k changefeeds took **several hours** to cancel, batched in groups of 10k, due to lock contention on claim updates.
- Large-cluster deployments (80+ nodes) have hit the same class of contention.
- TTL and stats jobs at 1M tables would produce so much churn that operator visibility into the jobs system degrades: *"a million jobs every five minutes is an unpleasant situation to be in if you're trying to figure out what was going on in this cluster."*
### Proposed approach
Replace the polling claim query with a **rangefeed** on `system.jobs`. Each node's registry listens to the table. When a claimable job appears, the node uses a **hashed-job-ID shard heuristic** for back-off:
- Compute `hash(job_id) mod numNodes` against the current instance cache.
- Zero back-off if the job hashes to "your" shard.
- ~30s back-off otherwise.
- Whichever node claims first wins; other nodes observe the claim via the rangefeed and drop their attempt.
This eliminates the per-tick contention storm without needing a dedicated leasing layer.
### Migration shape
The new adopter should be built **purely additive** on top of the existing pollers, gated by a cluster setting:
- V1 keeps the existing pollers running as a safety net, with adaptive back-off — *"if the last 10 polls you didn't see anything to do, start polling once every 10 minutes."*
- If the rangefeed gets stuck, restarting it is acceptable degradation: worst case "five minutes to start instead of eight seconds," covered by the poller backstop.
- In tests, pollers can be turned off (or metamorphically toggled).
- Phased default-on: optional + metamorphic in 26.3, try in cloud, default-on in a later release.
Epic CRDB-62562
Contributor guide
Assessment
This issue has not been assessed yet.