cockroachdb / cockroachdb/cockroach

kvserver: raft tick smearing still bursts scheduler wakeups within each pace step

Open
#169,962 1 comment 0 reactions 0 assignees View on GitHub
A-kv-replication C-enhancement O-agent O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Raft tick smearing (`e5a357f55bf`, v25.2.0) spreads enqueues across the
500ms tick interval in 1ms pacer steps. But each step still issues a burst
of `Cond.Signal()`/`Cond.Broadcast()` calls on `raftSchedulerShard`, making
~100 worker goroutines runnable within ~100µs. We have customer evidence
(v25.2.13) that this per-step herd is a recurring contributor to
sched-latency p99 breaches (≥2ms), even with smearing enabled.

**Walkthrough on a 50k-range store, 16+ vCPU, defaults**
(`RaftTickInterval=500ms`, `RaftTickSmearInterval=1ms`,
`defaultRaftSchedulerConcurrency=128`, `defaultRaftSchedulerShardSize=16`):

- pacer: `todo ≈ workLeft / 500` → ~100 ranges per 1ms step
- bucketed across 8 regular shards by hash → ~12 ranges/shard/step
- per shard: 12 < 16 → 12 `Signal()` calls
- across 8 shards: **~96 worker goroutines runnable within ~100µs every 1ms**

Customer trace (n18, `pprof_20260501_090528_18.pb.gz`):

> Heavy unblocker g18656 (108): `raftTickLoop → enqueueBatch → signal`
> Burst: 170 goroutines became runnable within ±1ms

`g18656` is the single tickLoop g; 108 wakeups attributed to one pace step
matches the math.

**Why the herd hurts even at ~100/step:**

1. 100 g's runnable within ~100µs is faster than the Go scheduler can
distribute across Ps.
2. Hog goroutines stretch the herd's tail. We measured hogs in customer
traces (CGO calloc 14.76ms, SQL optimizer 12.29ms, closedts sender
9.7ms, `Stores.TryLogFlowControlSendQueues` 12.58ms). When a herd lands
on a P held by a hog, queued workers wait until the hog yields or async
preempt fires (~10ms). p99 trivially crosses 2ms.
3. **Within-shard `Broadcast` cliff.** When `count >= numWorkers`,
`signal()` switches to `Broadcast()` — wakes all workers in the shard
regardless of work count. Hash collisions or load skew can trigger this
at the worst moment.
4. The tick herd compounds with `HandleRaftRequest → enqueue1`,
`enqueueRaftUpdateCheck → enqueue1`, and (in v25.2/25.3) closedts
sidetransport `Broadcast`. Tick alone is survivable; together they
push worker queues deep.

**Describe the solution you'd like**

Possible levers (none tried in production yet):

1. **Cap signals per pace step.** Instead of `signal(count_per_shard)`,
issue `min(count, K)` signals (K small, e.g. 4) and let each woken
worker drain multiple ranges from the queue.
2. **Avoid the `Broadcast` cliff.** Always use `Signal()` regardless of
`count >= numWorkers`. Workers self-wake from the queue when more work
arrives.
3. **Pace at the wakeup level, not the enqueue level.** The herd is on
`Cond.Signal`, not on enqueue. Pacing currently solves the macroscopic
500ms burst but not the microscopic 1ms burst.

**Describe alternatives you've considered**

- Lowering `RaftTickSmearInterval` below 1ms — hits `time.Ticker`/`time.Timer`
resolution and pacer overhead.
- Raising `RaftSchedulerShardSize` — reduces per-shard worker count, so the
`Broadcast` cliff triggers earlier.
- Lowering `RaftSchedulerShardSize` — more shards, more per-step signals,
same total work plus more lock acquisition overhead.

None of these address the fundamental issue that per-pace-step wakeups
happen synchronously within microseconds.

**Additional context**

- Related, fixed in v26.1+: #148211 (sidetransport pacing,
`081128daa76`) and `410a90c590d` (replace sidetransport `Broadcast` with
targeted `Signal`s). Same pattern, applied to a different subsystem.
- Code references:
- [raftScheduler.signal](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/scheduler.go) (`raftSchedulerShard.signal`, `enqueueBatch`)
- [raftTickLoop](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/store_raft.go) (`Store.raftTickLoop`)
- [taskpacer.Pace](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/taskpacer/pacer.go)
- Customer on v25.2.13. Smearing enabled. p99 sched-latency breaches
(≥2ms) recurring on n17 (heaviest), n18, n19 across the 190s pre-backup
window analyzed.

cc @iskettaneh

Epic: none

Jira issue: CRDB-63734

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.