cockroachdb / cockroachdb/cockroach
roachtest: improve observability around system health metrics
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Grey failures are notoriously hard to troubleshoot; a priori it's unknown whether a (workload) operation failed due to the database or because of an unavailability of system resource(s). Knowing exactly when a node experienced a system unavailability issue, e.g., network blip, is usually a valuable insight during a post-mortem. Fishing this information out of the system logs can be time-consuming.
We could configure external (network) probes [1]. Together with system and runtime metrics (via `node_exporter`), we could potentially derive a set of reliable "health metrics" to be included with the rest of the context around a test failure.
[1] https://github.com/prometheus/blackbox_exporter
Jira issue: CRDB-46931
Contributor guide
Assessment
This issue has not been assessed yet.