demml / demml/scouter

Require parameters for llm drift evals

Open
#107 0 comments 0 reactions 1 assignee Claimed by @thorrester View on GitHub
Dominant language
Rust
Stars
13
Forks
1
PR merge metrics
No merged PRs in 30d

Description

When implementing `LLMetric` as part of `LLMDriftProfile` logic, we'll need to make sure that we add logic to check for bound parameters in evaluation prompts. It should be expected that a combination of `input` or `output` parameters are provided in the prompt.

This is needed because when the server is running LLM evals with the user-defined workflow, it will inject the input and output into the prompts before calling and returning a metric.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.