cockroachdb / cockroachdb/cockroach

roachtest: improve observability around system health metrics

Open
#139,950 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Grey failures are notoriously hard to troubleshoot; a priori it's unknown whether a (workload) operation failed due to the database or because of an unavailability of system resource(s). Knowing exactly when a node experienced a system unavailability issue, e.g., network blip, is usually a valuable insight during a post-mortem. Fishing this information out of the system logs can be time-consuming.

We could configure external (network) probes [1]. Together with system and runtime metrics (via `node_exporter`), we could potentially derive a set of reliable "health metrics" to be included with the rest of the context around a test failure.

[1] https://github.com/prometheus/blackbox_exporter

Jira issue: CRDB-46931

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.