githubnext / githubnext/gh-aw-test

Suggested test: copilot/evals-model-alias

Closed
#9,065 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#47956 — propagate `ModelMappings` into evals `WorkflowData` to unblock AWF model alias resolution.

Evals jobs were producing 100% `UNKNOWN` answers because `buildEvalsEngineSteps` constructed `evalsData` without copying `ModelMappings` from the parent `WorkflowData`. This caused the AWF to exit with code 1 on an unresolvable model alias (e.g. `"small"`), silently discarded by `continue-on-error: true`. There is currently no E2E test that exercises the `evals:` frontmatter section at all, leaving this entire feature path untested.

## Proposed test

- **Workflow file**: `test-copilot-evals-frontmatter.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue` (the eval result is checked via the evals artifact, not a safe output)
- **Variant**: standard

## Minimal test prompt sketch

A minimal workflow with an `evals:` block in the frontmatter that uses a model alias (e.g. `model: small`). The agent is asked a fixed factual question. The test asserts that the evals job produces an `evals.jsonl` artifact (i.e. exit 0, not the silent failure from the alias-resolution bug) and that the overall workflow run succeeds.

## New fixtures or secrets needed

None beyond existing engine credentials. The `evals:` feature uses the standard Copilot engine path.

## Notes

There is an open suggestion `Suggested test: copilot/evals-frontmatter` already in `open-suggestions.json`. This issue provides the specific motivation from PR #47956 and the model-alias regression angle. If the earlier suggestion is already being acted on, this can be closed as a duplicate — but the model-alias regression angle (using `model: small`) should be incorporated into any evals test.

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/30152285891) · sonnet46 · 32.7 AIC · ⌖ 8.96 AIC · ⊞ 6.1K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the proposed test file, test-copilot-evals-frontmatter.md, and review the existing open-suggestions.json entry for copilot/evals-frontmatter. Define the workflow_dispatch case with the copilot engine, evals frontmatter using model: small, and create-issue safe output; done means the run succeeds and its evals artifact contains evals.jsonl.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.