githubnext / githubnext/gh-aw-test
Suggested test: copilot/report-incomplete-job-failure
- Dominant language
- Shell
- Stars
- 8
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
github/gh-aw#59316 (fixes github/gh-aw#59274) — `report_incomplete` now calls `core.setFailed()` so the Actions job conclusion becomes `failure`, even when issue reporting is disabled or fails. Previously the job stayed green.
## Proposed test
- **Workflow file**: existing `test-copilot-report-incomplete.md` needs its expected outcome updated, and/or a new variant `test-copilot-report-incomplete-create-issue-disabled.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `report-incomplete` with `create-issue: false`
- **Variant**: standard
## Minimal test prompt sketch
Agent calls `report_incomplete(reason: ...)` with `safe-outputs.report-incomplete.create-issue: false`. E2E harness should assert the workflow run **conclusion is `failure`**, not just that the run completed, and should NOT require an issue to be created.
## New fixtures or secrets needed
None. Requires updating `e2e.sh` validation logic (currently `trigger_workflow_dispatch_and_await_completion` / `wait_for_workflow` for the `report-incomplete` case appears to only check the run completed, not its conclusion) to assert `conclusion == "failure"` for this specific test case.
## Notes
The existing `test-copilot-report-incomplete.md` test currently expects/tolerates a green run. After gh-aw#59316 lands, that existing test's pass/fail semantics change — this should be reconciled (either update the existing test's expected conclusion, or add a distinct `create-issue:false` variant) before the harness reports false negatives.
> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/34205516021) · copilot · auto · 59.9 AIC · ⌖ 10.9 AIC · ⊞ 8.8K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with test-copilot-report-incomplete.md and inspect e2e.sh, especially trigger_workflow_dispatch_and_await_completion and wait_for_workflow. Decide whether to update the existing test or add the create-issue:false variant, then make validation require a failure conclusion without requiring an issue and run the relevant E2E test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, markdown, shell
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100