LLM Evaluation comparisons
Open
- Dominant language
- Rust
- Stars
- 40
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
run joint evaluations on two separate sets of LLM records
use case:
User has 2 versions of an api running and wants to do an A/B test
Contributor guide
Research direction
The issue provides no files, tests, or entry points. Begin by locating the existing LLM evaluation and record-set APIs, then define how two API versions are selected and what comparison output constitutes a completed A/B evaluation.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, backend-api-design, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100