microsoft / microsoft/Power-CAT-Copilot-Studio-Kit
Bug: Agent Review Tool - Instruction coverage reported as invocation-failed (GitHub Copilot harness)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 450
- Forks
- 95
- Avg merge
- 26m
- Merged PRs (30d)
- 5
Description
Description
Scope: GitHub Copilot harness agents.
The Agent Review Tool writes two components to cat_agentreviewjobitems per review: orchestration at cat_componentindex 9000 and coverage at 9001. Both complete successfully and both are stored with cat_itemstatus = Succeeded. The published review reflects the 9000 record but not the 9001 record, and reports instruction coverage as failed.
A review of a GitHub Copilot harness agent completed in 54 seconds. The agent node for instruction coverage returned a valid result: five elements (role, tone, boundaries, ambiguity, escalation), each with present: true and a verbatim evidence quote, and citationIds set to [{"id": "ms-authoring-instructions"}]. The following flow step wrote that result to cat_agentreviewjobitems and received HTTP 201.
The published review nevertheless reports:
analysisStatus.instructionCoverage.state = "failed"
analysisStatus.instructionCoverage.reason = "invocation-failed"
The UI shows the banner "AI instruction coverage failed. Deterministic instruction checks completed; rerun after the classifier is available."
Steps to reproduce
- Run an agent review on a GitHub Copilot harness agent.
- Query cat_agentreviewjobitems filtered on the run's cat_correlationid. Three rows are returned: the skill bundle at index 0, orchestration at 9000, and coverage at 9001. All three carry cat_itemstatus = Succeeded.
- Open cat_resultjson on the 9001 row and confirm it contains the complete five-element assessment.
- Open cat_reviewresultjson on the corresponding cat_agentreviews record and compare.
Expected behavior
The published review reads the coverage result from the 9001 record and reports the five criteria with their evidence as well, as it already does for the 9000 record.
Actual behavior
In the reviewed version, the "9000" component is read and reflected in the output while the "9001" component is not
Environment
Copilot Agent Kit, August 2026 release. agentVersion 2026-08-21. Agent type: GitHub Copilot harness. Reviewer model: GPT-5.5.
Additional context or logs
Impact: the banner leads a reader to conclude that role, tone, boundaries, ambiguity handling and escalation were not assessed, when all five were assessed and passed.
Correlation ID of the affected run available on request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how cat_agentreviewjobitems records are read when cat_componentindex is 9000 and 9001, then compare that path with cat_reviewresultjson on the corresponding cat_agentreviews record. Reproduce the Copilot harness review, verify the 9001 cat_resultjson contains the five-element assessment, and consider the issue done when the published review includes that coverage instead of reporting invocation-failed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100