kubernetes-sigs / kubernetes-sigs/devops-bench
feat(monitor): Add Performance Monitor to capture resource metrics across tasks
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
### Problem / Motivation
Beyond binary pass/fail outcomes, understanding the efficiency of agent execution (token consumption, latency, API call count, resource overhead) is critical for comparing models and practical deployment feasibility.
### Proposed Solution & Technical Details
- Implement a Performance Monitor module that wraps task execution.
- Capture telemetry:
- Total input/output token usage and cost estimations.
- Step count and wall-clock execution latency per task.
- Number of tool / kubectl invocations and command execution errors.
- Host CPU / memory utilization during benchmark runs.
- Output metrics into structured evaluation summary artifacts.
### Acceptance Criteria
- [ ] Performance monitor captures token, step, and latency metrics across all executed tasks.
- [ ] Structured performance statistics included in the final evaluation report JSON.
- [ ] Negligible overhead introduced to task execution runtime.
---
/kind feature
/area infra
/priority critical-urgent
Contributor guide
Assessment
This issue has not been assessed yet.