githubnext / githubnext/gh-aw-test

Suggested test: copilot/evals-frontmatter

Open
#5,793 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#43641 — feat: add BinEval evals support

PR #43641 adds a new top-level `evals:` frontmatter block. Workflows can now declare atomic binary YES/NO questions that are evaluated by an LLM after `safe_outputs` completes. Results are stored as JSONL artifacts and persisted to a git branch. No existing test exercises this new frontmatter feature.

## Proposed test

- **Workflow file**: `test-copilot-evals-frontmatter.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot
- **Safe output**: `create-issue`
- **Variant**: standard

## Minimal test prompt sketch

Ask the agent to create a GitHub issue with a fixed title (e.g. "evals smoke test"). Declare a single `evals:` question: "Did the agent create an issue?" Verify the compiled lockfile contains an `evals` job positioned after `safe_outputs`, and that the job runs to completion.

## New fixtures or secrets needed

None. The `evals` job calls the GitHub Models API using the runner token and persists results to a branch in `githubnext/gh-aw-test` — both already available.

## Notes

- Tests the shorthand `evals:` list form; the extended form (`evals.questions` + `engine-config.model` override) can be a follow-up.
- No overlap with existing tests — `test-copilot-create-issue.md` has no `evals:` block.
- Should eventually have nosandbox and siderepo variants, but the standard variant is the right starting point.

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `awmgmcpg`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "awmgmcpg"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/28782691319) · 34 AIC · ⌖ 6.61 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading test-copilot-create-issue.md and the gh-aw#43641 context, then add the proposed test workflow as test-copilot-evals-frontmatter.md with the standard copilot variant and workflow_dispatch trigger. Run the test and verify the compiled lockfile places an evals job after safe_outputs and that the job completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, shell, yaml
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.