githubnext / githubnext/gh-aw-test
Suggested test: copilot/evals-model-alias
- Dominant language
- Shell
- Stars
- 8
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
Link to the gh-aw PR: github/gh-aw#47956 — propagate `ModelMappings` into evals `WorkflowData` to unblock AWF model alias resolution.
Evals jobs were producing 100% `UNKNOWN` answers because `buildEvalsEngineSteps` constructed `evalsData` without copying `ModelMappings` from the parent `WorkflowData`. This caused the AWF to exit with code 1 on an unresolvable model alias (e.g. `"small"`), silently discarded by `continue-on-error: true`. There is currently no E2E test that exercises the `evals:` frontmatter section at all, leaving this entire feature path untested.
## Proposed test
- **Workflow file**: `test-copilot-evals-frontmatter.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue` (the eval result is checked via the evals artifact, not a safe output)
- **Variant**: standard
## Minimal test prompt sketch
A minimal workflow with an `evals:` block in the frontmatter that uses a model alias (e.g. `model: small`). The agent is asked a fixed factual question. The test asserts that the evals job produces an `evals.jsonl` artifact (i.e. exit 0, not the silent failure from the alias-resolution bug) and that the overall workflow run succeeds.
## New fixtures or secrets needed
None beyond existing engine credentials. The `evals:` feature uses the standard Copilot engine path.
## Notes
There is an open suggestion `Suggested test: copilot/evals-frontmatter` already in `open-suggestions.json`. This issue provides the specific motivation from PR #47956 and the model-alias regression angle. If the earlier suggestion is already being acted on, this can be closed as a duplicate — but the model-alias regression angle (using `model: small`) should be incorporated into any evals test.
> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/30152285891) · sonnet46 · 32.7 AIC · ⌖ 8.96 AIC · ⊞ 6.1K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the proposed test file, test-copilot-evals-frontmatter.md, and review the existing open-suggestions.json entry for copilot/evals-frontmatter. Define the workflow_dispatch case with the copilot engine, evals frontmatter using model: small, and create-issue safe output; done means the run succeeds and its evals artifact contains evals.jsonl.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100