[Feature] Add system metrics collected during evaluation to eval_output
Open
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
It would useful to collect system metrics, e.g. latency, during the evaluation and to provide a summary in the evaluation output.
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by locating where evaluation results are assembled into eval_output and where evaluation-time metrics are collected. Done requires an agreed set of system metrics and a defined summary in the evaluation output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100