cockroachdb / cockroachdb/cockroach
kvserver: observability for expired liveness lease
Open
A-kv-observability
C-enhancement
O-support
P-3
T-kv
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We've seen on multiple occasions (most recently in https://github.com/cockroachlabs/support/issues/3300) that an issue with the liveness range leaseholder can cause an cluster-wide outage. In such cases, we would like to be able to better understand what caused the lease to expire (e.g. CPU overload, disk issues).
Capturing an execution trace right before the lease is about to expire can help debug these situations better in the future. In the common case, the lease is extended after 3s, so if we were to capture an execution trace after 5s or so, this should only happen in the few cases where the lease is actually about to expire.
Jira issue: CRDB-50863
Contributor guide
Assessment
This issue has not been assessed yet.