microsoft / microsoft/Power-CAT-Copilot-Studio-Kit
Bug: Agent Review Tool - Review scores and finding evidence can be misleading (Standard Harness)
@Ramakrishnan24689 is already working on this.
Since Sep 18, 2026.
- Dominant language
- TypeScript
- Stars
- 450
- Forks
- 95
- Avg merge
- 26m
- Merged PRs (30d)
- 5
Description
Description
A score can be difficult to understand when completed, not-applicable, and unavailable checks are treated in the same way. Some findings also lack enough supporting detail.
Example
A test category contains four equal checks: one Pass, one Fail, one Not applicable, and one Not assessed. The completed applicable result is one Pass from two checks, which is 50%. The unavailable checks must not reduce this result to 25%. A missing-description finding should also identify the affected input, such as Ticket number.
Steps to reproduce
-
Create a test agent that produces the four result states.
-
Run an agent review.
-
Compare the category score, finding details, and review history.
Expected behavior
Calculate the score from completed applicable checks only. Exclude Not applicable and Not assessed checks. Show the affected item, the reason for the finding, and the available current value. If no check was completed, show the score as unavailable instead of 0%.
Actual behavior
In the reviewed version, excluded checks could make the score unclear. Some findings did not identify enough evidence, and an unavailable score could appear as 0% in review history.
Environment
No response
Additional context or logs
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.