bulk,admission: index backfill defaults are undersized for large nodes — 512 MiB backfiller buffer creates a synchronized L0 onset burst, and storage.max_compaction_concurrency=3 silently caps the elastic IO token economy

## Describe the problem

We ran an instrumented campaign of 48 identical online index builds (same `CREATE INDEX` on a
916 GB / 686M-row table, dropped between runs) on a 48-node / 96-store cluster of 64-vCPU machines
with NVMe stores, measuring lossless whole-build admission counters, per-store IO listener log
lines, and per-flush accounting (`buffering_adder=3,sst_batcher=1` vmodule). The campaign followed
two production incidents in which index backfills caused foreground REGULAR IO-token exhaustion
while the disks stayed far from saturation (sub-millisecond service times, queue depth ~3, on the
affected store throughout). Our starting point was both a duration and a safety problem: with the
protective throttle we had pinned after the first incident (`bulkio.ingest.sender_concurrency_limit
= 1`) and everything else at defaults, each build took **17–19 minutes** and still leaked
foreground token exhaustion at onset; under the posture described below the same build runs in
**4.0–4.5 minutes** with exactly 0 µs of foreground token exhaustion, at the default sender limit.

We root-caused the foreground damage to two defaults that don't scale to large nodes, plus
admission calibration that is very conservative for modern NVMe:

**1. `schemachanger.backfiller.max_buffer_size` = 512 MiB fragments the ingest and synchronizes the
onset burst.** With per-flush accounting on, every one of a build's 257 flushes buffered to exactly
the 512 MiB cap, and each flush was cut into ~240 SSTs at *range boundaries* (99.2% of all cuts —
the 16 MiB payload cap is irrelevant on a large table), for ~62,000 SSTs per build averaging
1.6 MiB. Worse, the cap synchronizes ~94 adders' first flushes into a multi-store AddSSTable burst
in the first seconds of the build — several stores flash from ~3 to 11–62 (once 157) sublevels in
under 15 seconds, faster than every reactive loop (admission's 15 s token recomputation, compaction,
scatter). At 2 GiB the cap stops binding (each adder's ~1.1 GiB share fits in one buffer): 96
flushes, ~24,000 SSTs, no onset burst at all, and builds ~2× faster. The correct sizing model is
bytes-per-adder-share, which suggests the setting could be auto-sized. The setting is non-public
with no sizing guidance we could find.

**2. `storage.max_compaction_concurrency` effectively defaults to min(3, CPUs−1) = 3, which
silently caps the whole bulk economy on big machines.** Admission control's elastic IO token budget
is computed from *measured compaction bandwidth out of L0*, so a 3-wide compaction ceiling on a
64-vCPU box with idle disks caps how much background work the system will ever admit — and lets
compaction debt pool (one production store had accumulated 1.44 TB of pending compaction, which a
3-wide ceiling drains at a trickle while 60+ cores idle; that debt store was the amplifier in our
August incident). Raising the ceiling raised the elastic budget mechanically: 3→8→16 cut
whole-build elastic admission waiting from 124,352 s to 30–57k s to 13,223 s and build time from
9.2 to 4.7 minutes (measured from the 2 GiB-buffer, sender=1 baseline — the campaign as a whole
went 17–19 minutes → 4.0–4.5), foreground untouched; 32 bought nothing (knee at 16). A CPU-proportional
default would have done this out of the box.

**3. With those two fixed, default admission calibration is the remaining tax — measured as a
controlled A/B.** Identical posture and workload, only `admission.l0_sub_level_count_overload_threshold`
at default 20 vs 40: the default ran the build 2× slower (8.3 vs 4.0–4.5 min) with 6.8 s of
cumulative foreground token starvation, 98.5 s of real foreground queueing, 10× the elastic
queueing, and 8 MiB grant floors across 39 stores — at a gauge L0 maximum of **12 sublevels** and
idle disks. Token limiting begins at threshold/4 = 5 sublevels at default
(`pkg/util/admission/io_load_listener.go`), which on NVMe that stays sub-millisecond at 60+
sublevels taxes foreground for L0 states the disk doesn't feel. A separate single-node
dose-response experiment (holding L0 at 0/5/10/20/40 mean sublevels under a point-read workload)
put numbers on the physics: with admission out of the way, 0→40 sublevels moved read p99 only
2.2→6.0 ms, while default admission at the same L0 produced 21 ms — the damage at double-digit
sublevels is calibration, not read amplification, on fast disks.

A corollary worth stating for #148371: after fixing the two defaults, **every legacy throttle
became unnecessary for us** — `bulkio.ingest.sender_concurrency_limit` at 1, 4, and default all
measured exactly 0 µs of foreground token exhaustion (we had been pinning it at 1 as a compensator
since July, and have retired the pin). We don't oppose removing the legacy throttles, but operators
who depend on them today are likely compensating for these defaults, and the removal will land
hardest if the defaults aren't fixed in the same release.

## To Reproduce

Large-node cluster (tested: 48 × m8gn.16xlarge, 2 NVMe stores each, v25.4.10), a table large
enough that backfill buffers hit the 512 MiB cap, `CREATE INDEX` under moderate foreground load.
Observe: (a) synchronized multi-store L0 sublevel spike within seconds of the backfill's first
flush; (b) starvation-class grants and REGULAR token exhaustion in the io_load_listener lines for
the interval after the burst; (c) whole-build elastic admission wait dominated by the compaction
ceiling (raise `storage.max_compaction_concurrency` and watch the elastic budget and build time
move mechanically). Full per-run matrix (48 builds), listener-log extracts, and the single-node L0
experiment are attached to support ticket 32238 and available on request.

## Expected behavior

1. `schemachanger.backfiller.max_buffer_size` sized per adder share (or a larger default for
machines with memory to spare — measured peak bulk memory at 2 GiB was 3.78 GiB on the busiest
node, against 256 GB boxes), so a large table's backfill doesn't open with a synchronized
multi-store SST burst that no reactive control can catch. Failing that: make the setting public
with sizing guidance.
2. A CPU-proportional `storage.max_compaction_concurrency` default (or auto-sizing), given its
role as the input to the elastic IO token budget.
3. Hardware-calibrated (or auto-calibrated) `admission.l0_sub_level_count_overload_threshold` —
the A/B above is direct evidence that the default calibration heavily taxes fast-NVMe fleets.
4. If the 26.2 distributed-merge backfill pipeline natively addresses the fragmentation and onset
behavior (it bypasses the BulkAdder buffer settings entirely), guidance on that timeline would
reshape 1.

## Environment

- CockroachDB v25.4.10, self-hosted on AWS EC2 (m8gn.16xlarge, 64 vCPU, 2 NVMe stores per node)
- 48 data nodes / 96 stores; test table 916 GB / 686M rows; RF=5
- Campaign settings landed on: `schemachanger.backfiller.max_buffer_size='2GiB'`,
`storage.max_compaction_concurrency=8` (16 as an off-peak build boost),
`admission.l0_sub_level_count_overload_threshold=40`,
`bulkio.index_backfill.elastic_control.enabled=true` (26.2 default, pre-adopted), everything
else default — all values re-verified foreground-clean under the final posture

## Additional context

- Support ticket 32238 carries the full detail paper (incident timelines, the 48-run matrix, flush
anatomy, the A/B, the single-node L0 experiment) and raw data.
- Settings we tested and found ineffective for the onset burst, for the record:
`kv.bulk_io_write.concurrent_addsstable_requests` (fails at every dose — onset damage tracks
files-per-range, not evaluation concurrency), `bulkio.ingest.flush_delay` (inert; async flushing
absorbs it), `bulkio.index_backfill.initial_splits_per_processor` (silently no-ops on large
tables at default ingest concurrency: 85 of 94 adders created zero split spans),
`pebble.pre_ingest_delay.*` (worked as a bridge at small buffers, redundant once the buffer is
sized right — consistent with its 26.1 removal).
- One asymmetry worth a look while removing throttles per #148371: the sender limiter is one shared
limiter per node acquired per-SST across callers — we verified index backfill; IMPORT/RESTORE/CDC
initial-scan callers may still need a burst control after removal if their buffer sizing differs.

Jira issue: CRDB-67745

Epic CRDB-65516

Start with pkg/util/admission/io_load_listener.go, then trace the three named settings: schemachanger.backfiller.max_buffer_size, storage.max_compaction_concurrency, and admission.l0_sub_level_count_overload_threshold. Review how their defaults affect index backfill admission on large NVMe nodes. Done means an agreed, validated scope for default changes or sizing guidance, including whether distributed merge changes the buffer issue.

Written by the indexing model from the issue text.