mattpocock / mattpocock/evalite
Track latency and allow passing to scorers and graphs
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Super important for certain use cases. Tracking latency of different models can be a key metric.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the model evaluation flow, scorer interfaces, and graph data handling in the TypeScript codebase. Determine where latency should be captured and how it can be passed through to scorers and graphs; done means latency is tracked for model runs and is available in both places.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, data-visualization, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100