microsoft / microsoft/Power-CAT-Copilot-Studio-Kit
Add non-binary metrics in addition to pass/fail result
Open
@psimolin is already working on this.
Since Aug 13, 2025.
enhancement
- Dominant language
- TypeScript
- Stars
- 450
- Forks
- 95
- Avg merge
- 26m
- Merged PRs (30d)
- 5
Description
Currently each test case reports a pass/fail result. However, we know that evaluation of generative answers can generate non-binary scoring. For example a score of 1-10 on different metrics such as relevance, truth, completeness, fluency, coherence, equivalence, and groundness, etc.
reporting on and tracking these non-binary metrics is critical to spot improvements and regressions between agent releases.
Example:
- My agent v1 completes test case 1 with result "Success"
- We make a change and we publish my agent v2
- We run test case 1 against my agent v2 and it still we get "Success" because the answer still satisfies the eval. However, the change that we made decreases the quality of the answer slightly (but not enough to make it fail).
Here we are with an undetected regression and no signal to show tat in the test run.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.