microsoft / microsoft/Power-CAT-Copilot-Studio-Kit

Add non-binary metrics in addition to pass/fail result

Open
#276 1 comment 0 reactions 1 assignee View on GitHub

@psimolin is already working on this.

Since Aug 13, 2025.

enhancement
Dominant language
TypeScript
Stars
450
Forks
95
Avg merge
26m
Merged PRs (30d)
5

Description

Currently each test case reports a pass/fail result. However, we know that evaluation of generative answers can generate non-binary scoring. For example a score of 1-10 on different metrics such as relevance, truth, completeness, fluency, coherence, equivalence, and groundness, etc.
reporting on and tracking these non-binary metrics is critical to spot improvements and regressions between agent releases.
Example:

  1. My agent v1 completes test case 1 with result "Success"
  2. We make a change and we publish my agent v2
  3. We run test case 1 against my agent v2 and it still we get "Success" because the answer still satisfies the eval. However, the change that we made decreases the quality of the answer slightly (but not enough to make it fail).

Here we are with an undetected regression and no signal to show tat in the test run.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.