githubnext / githubnext/gh-aw-test

Suggested test: copilot/partial-execution-after-safe-output

Closed
#7,954 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#46493 — fix: treat partial_execution + terminal safe-output as success-with-late-activity

Before this fix, if the Copilot agent continued issuing bash commands after emitting its terminal safe-output, the 20s post-result watchdog killed the process with SIGTERM → exit 1, which the harness classified as `partial_execution` and retried 4×. The fix makes the harness treat this case as success. There is no regression test.

## Proposed test

- **Workflow file**: `test-copilot-partial-execution-after-safe-output.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue`
- **Variant**: standard

## Minimal test prompt sketch

Prompt the agent to create an issue with a known title, then explicitly instruct it to run a few additional shell commands (e.g. `echo done`, `sleep 5`) after creating the issue. The harness should record the issue creation as the terminal safe-output and treat the subsequent watchdog-terminated activity as success rather than retrying as `partial_execution`.

## New fixtures or secrets needed

None. Uses standard `githubnext/gh-aw-test`.

## Notes

Existing `test-copilot-report-incomplete` tests the failure path; this would test the corresponding success path for post-safe-output activity. A `samples:` block can approximate the scenario for `--use-samples` mode, but the real regression value is in the live engine path where the agent naturally continues issuing exploratory commands after writing its output.

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `awmgmcpg`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "awmgmcpg"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/29681023308) · 27.4 AIC · ⌖ 5.44 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing test-copilot-report-incomplete test and the proposed test-copilot-partial-execution-after-safe-output.md workflow. Review its workflow_dispatch trigger, copilot engine, create-issue safe output, and standard variant, then run the test in samples mode and against the live engine. Done means the issue is created once and post-safe-output activity is treated as success without partial_execution retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell
Domain
ci-cd, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.