cockroachdb / cockroachdb/cockroach
kvserver: avoid frequent expensive scan on liveness lease extension
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The scan behind `MaybeGossipNodeLivenessRaftMuLocked` can be expensive[^1]. It happens under the liveness leaseholder's raftMu, which is bad. We do this every time the lease is extended, which seems overkill - doing it only on lease changes seems sufficient, especially since we are shortening the lease interval in 23.1. We should do a few immediate things here
- [ ] add simple instrumentation around the scan that logs when it's slow (backport)
- [ ] ideally perform the scan with iterator stats so that we can print them when it's slow (backport)
- [ ] only trigger the scan on leaseholder changes, but not on each extension (the TTL is infinity anyway!)
- [ ] write a KB
- [ ] can we avoid raftMu for this scan? That should be easy
In the longer run, we'll want to avoid the critical role of the liveness range in the first place, which is being explored through issues such as https://github.com/cockroachdb/cockroach/issues/93903. (@erikgrinaker have some better x-refs?)
https://github.com/cockroachdb/cockroach/blob/baf3469989b487c017723be49d366a2d6e86cb9e/pkg/kv/kvserver/replica_proposal.go#L433-L455
[^1]: with a 5m GC TTL, we saw it burn ~1s of CPU each lease interval, and sometimes more, causing failed liveness heartbeats. x-ref https://github.com/cockroachlabs/support/issues/2107
Jira issue: CRDB-24990
Contributor guide
Assessment
This issue has not been assessed yet.