Custom Evaluation
Open
- Dominant language
- Rust
- Stars
- 13
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Expand GenAI evals to allow for more robust evaluation types
Including:
- LLM as a judge (LLMJudgeTask)
- Structured output validation (FieldValidators)
- Matching validators (validating specific outputs types, e.g. string)
Functionality:
- Allow users to specify multiple evaluations for a single `event`
- Global test suite validations along with individualized assertions
- GenAI service profile type that allows users to validate multiple genai profiles at once
- Some genai applications will have more than one prompt. A user may want to build a test suite for each individual task as well as a global service evaluation task.
Contributor guide
Assessment
This issue has not been assessed yet.