cockroachdb / cockroachdb/cockroach
kv: tolerate disk stalls on otherwise healthy leaseholder VMs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We are increasingly seeing cases where a node's disk stalls while its VM remains healthy. This is problematic for range leases on that VM, because access to the VMs disk will stall and wedge traffic. It is problematic for two reasons:
- traffic will stall on the node while it still holds leases (6 seconds)
- traffic may remain stalled on the node even after it has given up its leases, until the disk stall detector kicks in a kills the process (20 seconds)
The second problem is being addressed by https://github.com/cockroachdb/cockroach/issues/104262.
The first problem is more difficult to solve. There are a few proposed solutions:
1. request hedging for read-only requests (https://github.com/cockroachdb/cockroach/issues/109320)
2. transfer leases away before they expire (https://cockroachlabs.slack.com/archives/C04G7B7FZUJ/p1706040946396869)
3. evaluate on the leaseholder using a follower replica's disk (https://cockroachlabs.slack.com/archives/C013M2PD35H/p1703257144945909?thread_ts=1702074693.889219&cid=C013M2PD35H)
This issue tracks the second two possible solutions.
Jira issue: CRDB-36180
Contributor guide
Assessment
This issue has not been assessed yet.