githubnext / githubnext/gh-aw-test

Suggested test: copilot/suppress-spurious-report-incomplete

Closed
#7,258 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#45599 — Suppress spurious `report_incomplete` failures after successful Copilot task output

A bug caused workflows to be marked incomplete even when Copilot completed its task and emitted safe outputs — if `task_complete` was not separately registered. The fix adds actuation-tracking to prevent this false negative.

## Proposed test

- **Workflow file**: `test-copilot-suppress-spurious-incomplete.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `add-comment` (then assert no `report-incomplete` is also emitted)
- **Variant**: standard

## Minimal test prompt sketch

Prompt the agent to post a comment on a known open issue. The workflow should succeed with exactly one `add-comment` safe output and zero `report-incomplete` outputs. This is a regression test for the scenario where the task-complete signal was dropped but the safe output was already written.

## New fixtures or secrets needed

None — requires an open issue in githubnext/gh-aw-test (already exists as a fixture pattern).

## Notes

The existing `test-copilot-report-incomplete` tests the intentional case where an agent deliberately calls `report_incomplete`. This proposed test covers the *unintentional* case — a successful completion that should never emit `report-incomplete`. The E2E harness would need to assert that `report-incomplete` is absent from the safe-outputs artifact.

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `awmgmcpg`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "awmgmcpg"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/29403625307) · 30.2 AIC · ⌖ 5.72 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the existing `test-copilot-report-incomplete` test and the E2E harness conventions for `test-copilot-suppress-spurious-incomplete.md`. Add the `workflow_dispatch` Copilot workflow using the existing open-issue fixture pattern, then verify the safe-outputs artifact contains exactly one `add-comment` and no `report-incomplete` output.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.