LLM experiment evaluations
Open
enhancement
ideation
- Dominant language
- Rust
- Stars
- 40
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
Use case: Api running different versions of prompts and/or providers. User is interested in comparing inputs and outputs and evaluating offline with LLM as a judge workflow
Things to consider:
- Consider an optimal approach, which may enforce some inflexibility on the user
- Escape hatch to allow users to generate metrics as they see fit.
- Do we implement t-test statistic calculation for comparisons?
- How do we record?
- DB migrations to the experiment metric table?
Contributor guide
Assessment
This issue has not been assessed yet.