cockroachdb / cockroachdb/cockroach

kvserver: avoid frequent expensive scan on liveness lease extension

Open
#97,966 6 comments 0 reactions 0 assignees View on GitHub
C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The scan behind `MaybeGossipNodeLivenessRaftMuLocked` can be expensive[^1]. It happens under the liveness leaseholder's raftMu, which is bad. We do this every time the lease is extended, which seems overkill - doing it only on lease changes seems sufficient, especially since we are shortening the lease interval in 23.1. We should do a few immediate things here

- [ ] add simple instrumentation around the scan that logs when it's slow (backport)
- [ ] ideally perform the scan with iterator stats so that we can print them when it's slow (backport)
- [ ] only trigger the scan on leaseholder changes, but not on each extension (the TTL is infinity anyway!)
- [ ] write a KB
- [ ] can we avoid raftMu for this scan? That should be easy

In the longer run, we'll want to avoid the critical role of the liveness range in the first place, which is being explored through issues such as https://github.com/cockroachdb/cockroach/issues/93903. (@erikgrinaker have some better x-refs?)

https://github.com/cockroachdb/cockroach/blob/baf3469989b487c017723be49d366a2d6e86cb9e/pkg/kv/kvserver/replica_proposal.go#L433-L455

[^1]: with a 5m GC TTL, we saw it burn ~1s of CPU each lease interval, and sometimes more, causing failed liveness heartbeats. x-ref https://github.com/cockroachlabs/support/issues/2107

Jira issue: CRDB-24990

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.