githubnext / githubnext/gh-aw-test

Suggested test: copilot/strict-mode-compile-zizmor-regression

Closed
#14,028 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

github/gh-aw#58339 "Fix zizmor 1.29 finding path parsing" — the compiler's strict-mode `zizmor` security-scan integration silently discarded all findings under zizmor 1.29 (schema changed `given_path` to `verbatim_path`), causing strict compilation to falsely report "0 warnings/errors". No E2E test exercises strict-mode compile/lint success reporting.

## Proposed test

- **Workflow file**: `test-copilot-strict-mode-compile.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue` (report compile/lint result)
- **Variant**: standard

## Minimal test prompt sketch

Have the workflow run `gh aw compile --strict` (or equivalent) against a bundled sample workflow and have the agent report success/failure of the strict compile step, ensuring the harness itself would catch a regression where strict mode silently no-ops.

## New fixtures or secrets needed

May need a local copy of `gh-aw` binary/CLI in the runner if not already available via `copilot-setup-steps.yml`; verify `gh aw` extension is installed in the test runner image.

## Notes

This is more of a meta/tooling regression test than a safe-output feature test — flagging as an issue rather than a PR since it may require a different harness shape (invoking the gh-aw CLI itself, not just an agent). Feasibility of running `zizmor`/strict-mode inside a `workflow_dispatch`-triggered copilot job needs triage.

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/33854350934) · copilot · auto · 41 AIC · ⌖ 12.2 AIC · ⊞ 8.8K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with test-copilot-strict-mode-compile.md and copilot-setup-steps.yml, verifying whether the gh-aw CLI is installed in the workflow-dispatch runner. Check how a bundled sample workflow can run gh aw compile --strict and report its result through the copilot create-issue output. Done means the E2E test exercises strict compilation and would fail rather than report success when zizmor findings are silently discarded.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell
Domain
ci-cd, devtools, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.