cockroachdb / cockroachdb/cockroach
storage: low level disk health monitoring daemon
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Pebble has a health checking VFS implementation, which typically sits at the top of the VFS / FS stack. This does a reasonable job of detecting disk stalls (either at the File level, or FS level). Events are propagated up to Cockroach, which can take the necessary action (log N times before crashing the node, etc.).
It's possible for our disk-health checking FS to detect software issues (eg, deadlocks, excessively slow encryption-at-rest metadata updates, etc). However, after the fact it can be hard to differentiate post-crash.
This proposal is for improved observability in this area, ideally operating at a lower level than the existing VFS / FS health checks.
**Describe the solution you'd like**
One implementation (with a prototype [here](https://gist.github.com/sean-/04d6c2e32b8c9e8f27d83d00a3750f71)) might involve creating a separate process that could be run on a node that could probe the block device backing a store. Such a tool could also be bundled into the Cockroach binary, for ease of distribution (e.g. `cockroach debug pebble vfs-monitor ...`, etc.).
The implementation could also be run as part of Cockroach itself (i.e. a monitor goroutine spawned by Cockroach itself, one for each store, etc.).
**Additional context**
More internal discussion [here](https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1673969932859699).
Jira issue: CRDB-23527
Contributor guide
Assessment
This issue has not been assessed yet.