github / github/gh-aw

Graders should include native Copilot tool calls in the automatic trace payload

Closed
#61,178 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 48m
Merged PRs (30d)
773

Description

## Problem / motivation

The deterministic grader runtime in `v0.88.7` builds `trace.toolCalls` from MCP gateway logs in `actions/setup/js/trace_graders.cjs`. Native Copilot tools recorded in staged `events.jsonl` files—such as the built-in `skill` tool—are therefore absent from the automatic grader payload.

This prevents `skill-constraint-coverage` from evaluating constraints against actual skill invocations. Although custom payloads can supply `agentOutput.toolCalls`, `pkg/workflow/compiler_yaml_post_agent.go` emits the graders step before user post-steps, so a post-step cannot normalize native events before automatic grading runs.

## Expected behavior or outcome

Automatic grader payloads should expose native agent tool calls with their names, arguments, call IDs, and correlated completion status. A repository should not need a separate post-run evaluator merely to make Copilot tool activity visible to gh-aw graders.

## Scope

- Reuse the existing Copilot `events.jsonl` parsing support.
- Normalize `tool.execution_start` and `tool.execution_complete` into grader `toolCalls`.
- Preserve arguments, including the target passed to the built-in `skill` tool.
- Avoid double-counting calls represented in both native and MCP logs.
- Alternatively, provide a documented pre-grader hook through which workflows can populate the normalized payload.

## Acceptance criteria

- [ ] A native `tool.execution_start` produces a `trace.toolCalls` entry containing `name`, `arguments`, and `toolCallId`.
- [ ] A matching `tool.execution_complete` determines the entry’s `success` value through `toolCallId` correlation.
- [ ] Calls without a matching completion remain distinguishable from successful calls.
- [ ] Native and gateway-derived records are deduplicated deterministically.
- [ ] `skill-constraint-coverage` can match a successful invocation of a specific skill without repository-supplied grading logic.
- [ ] The unused agent-log path constants in `trace_graders.cjs` are either wired into preprocessing or removed.
- [ ] Tests cover arguments, failed and incomplete calls, correlation, and deduplication.

This is an enhancement to grader trace coverage rather than a regression; current behavior safely produces an unmet or not-applicable result.

Contributor guide

Open the contributing guide

Research direction

Start with actions/setup/js/trace_graders.cjs and the existing Copilot events.jsonl parsing support, then inspect how pkg/workflow/compiler_yaml_post_agent.go orders the graders step and post-steps. Trace native tool.execution_start and tool.execution_complete records into the grader payload, preserving arguments and correlation while deduplicating MCP records. Done means the listed acceptance cases are covered by tests and skill-constraint-coverage sees successful native skill calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, javascript
Domain
ci-cd, devtools, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.