mattpocock / mattpocock/evalite
Score calculation needs more explanation
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Would appreciate if the docs explain how the score of an eval is calculated when trialCount > 1.
I tested it myself and it averages the score from all runs, which for me is the expected behavior.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the documentation that describes eval score calculation and review the issue's stated behavior for trialCount greater than 1. Update the relevant explanation so it clearly states how scores from multiple runs are combined; done when the documented behavior matches the observed averaging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100