cockroachdb / cockroachdb/cockroach

storage: low level disk health monitoring daemon

Open
#95,469 0 comments 0 reactions 0 assignees View on GitHub
A-storage C-enhancement T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Pebble has a health checking VFS implementation, which typically sits at the top of the VFS / FS stack. This does a reasonable job of detecting disk stalls (either at the File level, or FS level). Events are propagated up to Cockroach, which can take the necessary action (log N times before crashing the node, etc.).

It's possible for our disk-health checking FS to detect software issues (eg, deadlocks, excessively slow encryption-at-rest metadata updates, etc). However, after the fact it can be hard to differentiate post-crash.

This proposal is for improved observability in this area, ideally operating at a lower level than the existing VFS / FS health checks.

**Describe the solution you'd like**

One implementation (with a prototype [here](https://gist.github.com/sean-/04d6c2e32b8c9e8f27d83d00a3750f71)) might involve creating a separate process that could be run on a node that could probe the block device backing a store. Such a tool could also be bundled into the Cockroach binary, for ease of distribution (e.g. `cockroach debug pebble vfs-monitor ...`, etc.).

The implementation could also be run as part of Cockroach itself (i.e. a monitor goroutine spawned by Cockroach itself, one for each store, etc.).

**Additional context**

More internal discussion [here](https://cockroachlabs.slack.com/archives/CAC6K3SLU/p1673969932859699).

Jira issue: CRDB-23527

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.