kubernetes-sigs / kubernetes-sigs/devops-bench

feat(monitor): Add Performance Monitor to capture resource metrics across tasks

Open
#97 3 comments 0 reactions 1 assignee Claimed by @eugeneng04 View on GitHub
kind/feature priority/critical-urgent
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

### Problem / Motivation
Beyond binary pass/fail outcomes, understanding the efficiency of agent execution (token consumption, latency, API call count, resource overhead) is critical for comparing models and practical deployment feasibility.

### Proposed Solution & Technical Details
- Implement a Performance Monitor module that wraps task execution.
- Capture telemetry:
- Total input/output token usage and cost estimations.
- Step count and wall-clock execution latency per task.
- Number of tool / kubectl invocations and command execution errors.
- Host CPU / memory utilization during benchmark runs.
- Output metrics into structured evaluation summary artifacts.

### Acceptance Criteria
- [ ] Performance monitor captures token, step, and latency metrics across all executed tasks.
- [ ] Structured performance statistics included in the final evaluation report JSON.
- [ ] Negligible overhead introduced to task execution runtime.

---
/kind feature
/area infra
/priority critical-urgent

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.