githubnext / githubnext/gh-aw-test

Suggested test: copilot/agent-continue-on-error

Open
#15,236 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#60350 — the compiler previously accepted `jobs.agent.continue-on-error` in frontmatter but silently dropped it from the generated workflow. It now propagates the explicit boolean to the generated `agent` job.

## Proposed test

- **Workflow file**: `test-copilot-agent-continue-on-error.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `noop` (or `missing-tool`) so the run still emits a benign safe output when the agent step is forced to fail
- **Variant**: standard

## Minimal test prompt sketch

Set `jobs.agent.continue-on-error: true` in frontmatter and have the prompt intentionally invoke a tool/command guaranteed to fail (e.g. reference a nonexistent MCP tool), then still call the `noop` safe output afterward. Assert via the compiled lock file (or run conclusion) that the `agent` job carries `continue-on-error: true` and the overall run conclusion is not `failure` despite the failing step.

## New fixtures or secrets needed

None.

## Notes

This is closer to a compiler/lockfile assertion than a behavioral one, since the E2E harness generally checks run conclusion. Verifying "job succeeded despite an internal failing step" may need a dedicated assertion in `e2e.sh` (checking job-level `conclusion` via `gh run view --json jobs`) rather than the existing overall-run-conclusion check — flagging as an open question for the harness maintainer to confirm feasibility before converting to a PR.

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/34747998916) · copilot · auto · 71.7 AIC · ⌖ 5.41 AIC · ⊞ 8.4K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the proposed test fixture, test-copilot-agent-continue-on-error.md, and inspect e2e.sh plus the compiled lock-file assertions. Check whether gh run view --json jobs can verify the agent job conclusion; done means the generated job preserves continue-on-error: true and the intentionally failing run is not reported as a failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, markdown, shell
Domain
ci-cd, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.