microsoft / microsoft/Power-CAT-Copilot-Studio-Kit

Bug: Agent Review Tool - Review scores and finding evidence can be misleading (Standard Harness)

Open
#874 0 comments 0 reactions 1 assignee View on GitHub

@Ramakrishnan24689 is already working on this.

Since Sep 18, 2026.

bug
Dominant language
TypeScript
Stars
450
Forks
95
Avg merge
26m
Merged PRs (30d)
5

Description

Description

A score can be difficult to understand when completed, not-applicable, and unavailable checks are treated in the same way. Some findings also lack enough supporting detail.

Example

A test category contains four equal checks: one Pass, one Fail, one Not applicable, and one Not assessed. The completed applicable result is one Pass from two checks, which is 50%. The unavailable checks must not reduce this result to 25%. A missing-description finding should also identify the affected input, such as Ticket number.

Steps to reproduce
  • Create a test agent that produces the four result states.

  • Run an agent review.

  • Compare the category score, finding details, and review history.

Expected behavior

Calculate the score from completed applicable checks only. Exclude Not applicable and Not assessed checks. Show the affected item, the reason for the finding, and the available current value. If no check was completed, show the score as unavailable instead of 0%.

Actual behavior

In the reviewed version, excluded checks could make the score unclear. Some findings did not identify enough evidence, and an unavailable score could appear as 0% in review history.

Environment

No response

Additional context or logs

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.