githubnext / githubnext/gh-aw-test
Suggested test: copilot/strict-mode-compile-zizmor-regression
- Dominant language
- Shell
- Stars
- 8
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
github/gh-aw#58339 "Fix zizmor 1.29 finding path parsing" — the compiler's strict-mode `zizmor` security-scan integration silently discarded all findings under zizmor 1.29 (schema changed `given_path` to `verbatim_path`), causing strict compilation to falsely report "0 warnings/errors". No E2E test exercises strict-mode compile/lint success reporting.
## Proposed test
- **Workflow file**: `test-copilot-strict-mode-compile.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue` (report compile/lint result)
- **Variant**: standard
## Minimal test prompt sketch
Have the workflow run `gh aw compile --strict` (or equivalent) against a bundled sample workflow and have the agent report success/failure of the strict compile step, ensuring the harness itself would catch a regression where strict mode silently no-ops.
## New fixtures or secrets needed
May need a local copy of `gh-aw` binary/CLI in the runner if not already available via `copilot-setup-steps.yml`; verify `gh aw` extension is installed in the test runner image.
## Notes
This is more of a meta/tooling regression test than a safe-output feature test — flagging as an issue rather than a PR since it may require a different harness shape (invoking the gh-aw CLI itself, not just an agent). Feasibility of running `zizmor`/strict-mode inside a `workflow_dispatch`-triggered copilot job needs triage.
> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/33854350934) · copilot · auto · 41 AIC · ⌖ 12.2 AIC · ⊞ 8.8K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with test-copilot-strict-mode-compile.md and copilot-setup-steps.yml, verifying whether the gh-aw CLI is installed in the workflow-dispatch runner. Check how a bundled sample workflow can run gh aw compile --strict and report its result through the copilot create-issue output. Done means the E2E test exercises strict compilation and would fail rather than report success when zizmor findings are silently discarded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, shell
- Domain
- ci-cd, devtools, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100