microsoft / microsoft/Power-CAT-Copilot-Studio-Kit

Bug: Agent Review Tool - Instruction coverage reported as invocation-failed (GitHub Copilot harness)

Open
#879 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
450
Forks
95
Avg merge
26m
Merged PRs (30d)
5

Description

Description

Scope: GitHub Copilot harness agents.
The Agent Review Tool writes two components to cat_agentreviewjobitems per review: orchestration at cat_componentindex 9000 and coverage at 9001. Both complete successfully and both are stored with cat_itemstatus = Succeeded. The published review reflects the 9000 record but not the 9001 record, and reports instruction coverage as failed.

A review of a GitHub Copilot harness agent completed in 54 seconds. The agent node for instruction coverage returned a valid result: five elements (role, tone, boundaries, ambiguity, escalation), each with present: true and a verbatim evidence quote, and citationIds set to [{"id": "ms-authoring-instructions"}]. The following flow step wrote that result to cat_agentreviewjobitems and received HTTP 201.

The published review nevertheless reports:

analysisStatus.instructionCoverage.state = "failed"
analysisStatus.instructionCoverage.reason = "invocation-failed"

The UI shows the banner "AI instruction coverage failed. Deterministic instruction checks completed; rerun after the classifier is available."

Steps to reproduce
  1. Run an agent review on a GitHub Copilot harness agent.
  2. Query cat_agentreviewjobitems filtered on the run's cat_correlationid. Three rows are returned: the skill bundle at index 0, orchestration at 9000, and coverage at 9001. All three carry cat_itemstatus = Succeeded.
  3. Open cat_resultjson on the 9001 row and confirm it contains the complete five-element assessment.
  4. Open cat_reviewresultjson on the corresponding cat_agentreviews record and compare.
Expected behavior

The published review reads the coverage result from the 9001 record and reports the five criteria with their evidence as well, as it already does for the 9000 record.

Actual behavior

In the reviewed version, the "9000" component is read and reflected in the output while the "9001" component is not

Environment

Copilot Agent Kit, August 2026 release. agentVersion 2026-08-21. Agent type: GitHub Copilot harness. Reviewer model: GPT-5.5.

Additional context or logs

Impact: the banner leads a reader to conclude that role, tone, boundaries, ambiguity handling and escalation were not assessed, when all five were assessed and passed.

Correlation ID of the affected run available on request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how cat_agentreviewjobitems records are read when cat_componentindex is 9000 and 9001, then compare that path with cat_reviewresultjson on the corresponding cat_agentreviews record. Reproduce the Copilot harness review, verify the 9001 cat_resultjson contains the five-element assessment, and consider the issue done when the published review includes that coverage instead of reporting invocation-failed.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.