cockroachdb / cockroachdb/cockroach
roachprod/failure-injection: add cgroups metrics as a scrape target
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently our cgroups disk stall validation consists of sampling cgroups i/o by reading `/sys/fs/cgroup/system.slice/io.stat` and doing `grep`/`awk` parsing.
```
// Check the number of bytes read and written to disk.
res, err := s.RunWithDetails(
ctx, l, node,
fmt.Sprintf(`grep -E '%d:%d' /sys/fs/cgroup/system.slice/io.stat |`, maj, min),
`grep -oE 'rbytes=[0-9]+|wbytes=[0-9]+' |`,
`awk -F= '{printf "%s ", $2} END {print ""}'`,
)
```
Instead, we should investigate using https://github.com/arianvp/cgroup-exporter/tree/main/collector, i.e. adding it as a scrape target we can view/query from in Grafana. This would bring multiple benefits over the current approach:
1. We can have a continuous view of throughput instead of just the discrete intervals we decide to sample.
2. Grafana provides a better visual representation.
3. More generic implementation to scrape all cgroups metrics not just read/write throughput.
Jira issue: CRDB-49221
Contributor guide
Assessment
This issue has not been assessed yet.