Graders should include native Copilot tool calls in the automatic trace payload
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 541
- Avg merge
- 5h 48m
- Merged PRs (30d)
- 773
Description
## Problem / motivation
The deterministic grader runtime in `v0.88.7` builds `trace.toolCalls` from MCP gateway logs in `actions/setup/js/trace_graders.cjs`. Native Copilot tools recorded in staged `events.jsonl` files—such as the built-in `skill` tool—are therefore absent from the automatic grader payload.
This prevents `skill-constraint-coverage` from evaluating constraints against actual skill invocations. Although custom payloads can supply `agentOutput.toolCalls`, `pkg/workflow/compiler_yaml_post_agent.go` emits the graders step before user post-steps, so a post-step cannot normalize native events before automatic grading runs.
## Expected behavior or outcome
Automatic grader payloads should expose native agent tool calls with their names, arguments, call IDs, and correlated completion status. A repository should not need a separate post-run evaluator merely to make Copilot tool activity visible to gh-aw graders.
## Scope
- Reuse the existing Copilot `events.jsonl` parsing support.
- Normalize `tool.execution_start` and `tool.execution_complete` into grader `toolCalls`.
- Preserve arguments, including the target passed to the built-in `skill` tool.
- Avoid double-counting calls represented in both native and MCP logs.
- Alternatively, provide a documented pre-grader hook through which workflows can populate the normalized payload.
## Acceptance criteria
- [ ] A native `tool.execution_start` produces a `trace.toolCalls` entry containing `name`, `arguments`, and `toolCallId`.
- [ ] A matching `tool.execution_complete` determines the entry’s `success` value through `toolCallId` correlation.
- [ ] Calls without a matching completion remain distinguishable from successful calls.
- [ ] Native and gateway-derived records are deduplicated deterministically.
- [ ] `skill-constraint-coverage` can match a successful invocation of a specific skill without repository-supplied grading logic.
- [ ] The unused agent-log path constants in `trace_graders.cjs` are either wired into preprocessing or removed.
- [ ] Tests cover arguments, failed and incomplete calls, correlation, and deduplication.
This is an enhancement to grader trace coverage rather than a regression; current behavior safely produces an unmet or not-applicable result.
Contributor guide
Research direction
Start with actions/setup/js/trace_graders.cjs and the existing Copilot events.jsonl parsing support, then inspect how pkg/workflow/compiler_yaml_post_agent.go orders the graders step and post-steps. Trace native tool.execution_start and tool.execution_complete records into the grader payload, preserving arguments and correlation while deduplicating MCP records. Done means the listed acceptance cases are covered by tests and skill-constraint-coverage sees successful native skill calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, javascript
- Domain
- ci-cd, devtools, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100