kvserver: consistency checker defaults cannot complete the 24h cycle on high-ranges-per-store clusters, and the queue saturates invisibly (pending gauge clamps at 10,000)

## Describe the problem

On clusters with a high range count per store, the consistency checker's defaults combine into a
queue that is permanently saturated and structurally unable to meet
`server.consistency_check.interval`, and the operator cannot observe how far behind it actually is.
Four defaults/design choices interact:

1. **`server.consistency_check.max_rate` defaults to 8 MiB/s per store**, sized for an era of much
slower disks and far fewer replicas per store. On NVMe-backed stores this is orders of magnitude
below what the hardware can absorb.
2. **The rate limiter is per-store and shared**: `store.go` creates a single
`quotapool.RateLimiter` per store from `max_rate`, and the checksum computations a store performs
*as a follower for other stores' checks* draw from the same pool. The comment above the checksum
task in `replica_consistency.go` (release-25.4) notes that concurrent checks "share the rate
limit in r.store.consistencyLimiter, so if too many run at the same time, chances are they will
time out", with average incoming in-flight collections per node equal to the replication factor.
Per-check latency therefore scales with replicas-per-store, not just range size.
3. **The consistency queue processes one replica at a time per store** — `maxConcurrency` is left 0
in its `queueConfig`, which defaults to 1, and as far as we can tell there is no cluster setting
to raise it. Fleet-wide dequeue throughput is hard-capped at `stores / per-check-seconds`.
4. **`queue.consistency.pending` clamps at the base queue's max size (10,000 per store)**, so once a
cluster falls behind, the one gauge that reports the backlog reads a constant and the true
overdue count and the oldest-unchecked age are unobservable.

The structural arithmetic (derived, not measured): meeting the 24h default interval requires
`ranges / 86,400` checks/s; the queue can deliver at most `stores / per-check-seconds`. Because
per-check time grows with replicas-per-store (point 2), consolidating the same data onto fewer,
larger stores — the direction modern hardware pushes everyone — makes both sides of the inequality
worse at once.

## What we measured (production, v25.4.10)

Self-hosted AWS cluster, 48 data nodes (m8gn family, 2 NVMe-backed stores each, 96 data stores),
RF=5, 825,643 ranges / 4,128,654 replicas. In mid-August we completed a planned consolidation from
~380 stores onto these 96 (range count essentially flat, +6%). We had already raised `max_rate` to
128 MiB (16× default) in June. Prometheus-scraped CRDB metrics:

- Before consolidation (~380 stores): ~9.0 checks/s fleet-wide at ~1.9 s per check
(derived as `rate(processingnanos)/rate(process.success)`).
- After (96 stores): **~5.49/s at ~12.75 s per check** — a 6.7× per-check increase, consistent with
each store's follower checksum-scan share rising ~4× under the shared limiter. ~70 of 96 stores
are inside a check at any instant, i.e. at the concurrency ceiling.
- Required rate for the 24h interval: **9.56/s**. Hard ceiling at 12.75 s/check: **~7.5/s**. The
fleet is structurally unable to keep up regardless of backlog; the effective audit cycle is ~1.7
days at current settings.
- `queue.consistency.pending` has sat at **exactly 10,000 on every one of the 96 stores (~960,000
total) with zero variance for days** — the clamp, not the true overdue count.
- `queue.consistency.process.failure` runs ~59/day during the backlog vs ~0.15/day baseline, which
matches the timeout behavior the `replica_consistency.go` comment predicts under limiter
contention. No divergence evidence, no node fatals.
- A brief controlled test raising `max_rate` 128→164 MiB dropped per-check time from 12.75 s to at
most 8.7 s within the first hour (window still blended pre-change samples), i.e. the checker is
scan-bandwidth-bound at these settings on this geometry — the limiter, not the hardware, is the
binding constraint. Disk IO overload signals stayed at their noise floor throughout.

We also could not find any admission-control integration (elastic CPU or store IO tokens) on the
checksum scan path in release-25.4 — the digest computation reads a storage snapshot paced only by
that per-store rate limiter — so today `max_rate` is simultaneously the only foreground protection
and the throughput bottleneck.

## Expected behavior

A cluster running at defaults should either complete its consistency cycle within the configured
interval or tell the operator loudly that it can't. Concretely, in rough priority order:

1. **Scale-aware defaults or auto-sizing for `max_rate`** — e.g. derived from provisioned
bandwidth, store count, or replicas-per-store — or at minimum documented sizing guidance. The
8 MiB default is a rounding error on modern NVMe.
2. **Observability of the true backlog**: don't clamp `queue.consistency.pending` at the queue's
max size (or expose that clamping occurred), and export the oldest-unchecked-range age so
"effective audit cycle" is a monitorable SLO. An alert-worthy signal for "the cycle can no
longer complete at current settings" would catch this class of problem at onset.
3. **A supported way to raise per-store check concurrency**, or pace the checksum scans through
admission control (elastic) so the rate limiter can be opened up safely instead of serving as
both throttle and safety mechanism.
4. **Distinguishable failure causes**: `process.failure` mixes checksum-collection timeouts and
replica-movement races with anything that would actually indicate divergence; a timeout that is
a predictable consequence of limiter contention should be separable from a real problem.

## Environment

- CockroachDB v25.4.10, self-hosted on AWS EC2 (m8gn instance family, NVMe instance storage)
- 48 data nodes / 96 data stores / 825,643 ranges / RF=5
- `server.consistency_check.max_rate = 128 MiB` (raised from default in June),
`server.consistency_check.interval = 24h` (default)

## Additional context

#67004 describes the core arithmetic of this problem (default `max_rate` cannot keep up on even
moderately large deployments; queue permanently swollen; "causes support inquiries") and has been
open since 2021. This report confirms it at production scale and adds what we believe is new: the
shared-limiter mechanism that makes per-check time scale with replicas-per-store (so consolidation
onto bigger nodes makes it worse quadratically, not linearly), the 10,000 pending-gauge clamp that
hides the backlog, the measured failure-rate increase under limiter contention, and concrete asks.
#83190 (make `max_rate` private) is worth weighing against this evidence — today that setting is
the only lever an operator has when the cycle can't complete.

Support ticket 32292 (filed 2026-08-27) covers the operational side of this and has a debug zip and
raw tsdump for the affected window attached; we can supply the full metric pulls behind the numbers
above. This issue is the generalization: the defaults produce an invisible, permanent saturation on
any cluster whose ranges-per-store crosses the arithmetic above, and the consolidation that
triggered it for us is the ordinary trajectory of hardware upgrades.

Jira issue: CRDB-67744

Start by reading store.go for the per-store quotapool.RateLimiter and replica_consistency.go for checksum-task concurrency and timeout behavior. Trace queueConfig, maxConcurrency, and the queue.consistency.pending metric to confirm the throughput and backlog limits. Done requires a maintainer-agreed scope covering the selected default, concurrency, backlog, or failure-observability behavior, with validation against the reported metrics.

Written by the indexing model from the issue text.