Clean metrics computation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 324
Description
Use cases, pain points, and background
Right now metrics can be calculated per benchmark in any way (e.g. responses_api_agents/scicode_agent/app.py _across_run_stats) without any naming conventions nor structure (e.g. one function with clearly defined input per metric). This makes writing general code (e.g. for computing CIs) difficult.
We need a way to still provide flexibility but with clean rules.
Description:
TBD
Design:
TBD
Out of scope:
TBD
Acceptance Criteria:
- TBD
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing responses_api_agents/scicode_agent/app.py, especially the _across_run_stats implementation, and compare how metrics are computed across benchmarks. The issue leaves the design and acceptance criteria as TBD, so the metric interface, naming rules, and definition of done need to be clarified before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100