demml / demml/scouter

Custom Evaluation

Open
#148 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
13
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Expand GenAI evals to allow for more robust evaluation types

Including:

- LLM as a judge (LLMJudgeTask)
- Structured output validation (FieldValidators)
- Matching validators (validating specific outputs types, e.g. string)

Functionality:

- Allow users to specify multiple evaluations for a single `event`
- Global test suite validations along with individualized assertions
- GenAI service profile type that allows users to validate multiple genai profiles at once
- Some genai applications will have more than one prompt. A user may want to build a test suite for each individual task as well as a global service evaluation task.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.