cockroachdb / cockroachdb/cockroach

kv: tolerate disk stalls on otherwise healthy leaseholder VMs

Open
#119,407 0 comments 0 reactions 0 assignees View on GitHub
A-kv C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We are increasingly seeing cases where a node's disk stalls while its VM remains healthy. This is problematic for range leases on that VM, because access to the VMs disk will stall and wedge traffic. It is problematic for two reasons:
- traffic will stall on the node while it still holds leases (6 seconds)
- traffic may remain stalled on the node even after it has given up its leases, until the disk stall detector kicks in a kills the process (20 seconds)

The second problem is being addressed by https://github.com/cockroachdb/cockroach/issues/104262.

The first problem is more difficult to solve. There are a few proposed solutions:
1. request hedging for read-only requests (https://github.com/cockroachdb/cockroach/issues/109320)
2. transfer leases away before they expire (https://cockroachlabs.slack.com/archives/C04G7B7FZUJ/p1706040946396869)
3. evaluate on the leaseholder using a follower replica's disk (https://cockroachlabs.slack.com/archives/C013M2PD35H/p1703257144945909?thread_ts=1702074693.889219&cid=C013M2PD35H)

This issue tracks the second two possible solutions.

Jira issue: CRDB-36180

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.