agent-substrate / agent-substrate/substrate
[benchmarking] Expose host kernel PSI metrics and relabel root cgroup in cAdvisor
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 316
- Avg merge
- 2d 43m
- Merged PRs (30d)
- 287
Description
### Background & Problem
When benchmarking Agent Substrate under realistic agent workloads (running test suites, compilation, and git operations across concurrent actor sandboxes), diagnosing whether tail latency spikes and cold-start regressions stem from CPU throttling, memory thrashing, or disk I/O contention requires Linux kernel Pressure Stall Information (PSI).
Currently, cAdvisor in `benchmarking/monitoring.yaml` drops PSI metrics and leaves the root cgroup `id="/"` unlabeled, making it difficult to isolate host-level resource starvation from actor container pressure.
### Proposed Changes
1. Relabel the root cgroup `id="/"` to `container="node"` in the cAdvisor scrape job in `benchmarking/monitoring.yaml`.
2. Expand the metric allowlist to capture `container_pressure_{cpu,memory,io}_waiting_seconds_total`, along with network throughput and filesystem usage.
### References
- [Substrate Benchmark Telemetry & Metrics Enhancement](https://docs.google.com/document/d/1f9b1mTyxICeLZPojrxCujZ16Elh4IFo3_WmBeNCvk1c/edit?pli=1&tab=t.0)
- [Agent Substrate: Actor Density Benchmark Specs](https://docs.google.com/document/d/1sv5aBXvGOQ69iaqFxdPZ-d6tNbh5AMuwKODDyYjzYQw/edit?tab=t.0)
cc @maxsmythe @roycaihw - happy to submit a PR for this!
Contributor guide
Assessment
This issue has not been assessed yet.