cockroachdb / cockroachdb/cockroach

roachprod/failure-injection: add cgroups metrics as a scrape target

Open
#144,052 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently our cgroups disk stall validation consists of sampling cgroups i/o by reading `/sys/fs/cgroup/system.slice/io.stat` and doing `grep`/`awk` parsing.

```
// Check the number of bytes read and written to disk.
res, err := s.RunWithDetails(
ctx, l, node,
fmt.Sprintf(`grep -E '%d:%d' /sys/fs/cgroup/system.slice/io.stat |`, maj, min),
`grep -oE 'rbytes=[0-9]+|wbytes=[0-9]+' |`,
`awk -F= '{printf "%s ", $2} END {print ""}'`,
)
```

Instead, we should investigate using https://github.com/arianvp/cgroup-exporter/tree/main/collector, i.e. adding it as a scrape target we can view/query from in Grafana. This would bring multiple benefits over the current approach:

1. We can have a continuous view of throughput instead of just the discrete intervals we decide to sample.
2. Grafana provides a better visual representation.
3. More generic implementation to scrape all cgroups metrics not just read/write throughput.

Jira issue: CRDB-49221

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.