githubnext / githubnext/gh-aw-test

Suggested test: copilot/custom-engine-third-party

Open
#7,259 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#45515 — Remove built-in Crush engine support

Built-in Crush support was removed because Crush can now be implemented via the `engine.behaviors` custom engine import mechanism. PR #45559 also added docs for configuring third-party agents. There is currently no E2E test that exercises a custom (non-built-in) engine definition loaded via `engine.behaviors`.

## Proposed test

- **Workflow file**: `test-copilot-custom-engine-behaviors.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: copilot (with a custom `engine.behaviors` import overlay)
- **Safe output**: `create-issue`
- **Variant**: standard

## Minimal test prompt sketch

Define a minimal custom engine behavior override in the workflow frontmatter (e.g. override `agentTimeout` or a tool allowlist via `engine.behaviors`) and have the agent create an issue confirming the behavior was applied. This exercises the import/merge path for custom engine definitions without needing a new engine binary.

## New fixtures or secrets needed

None if the test only overrides frontmatter fields. If a real external engine binary is needed, that would require a new secret/fixture — but a frontmatter-only override should be sufficient to exercise the compilation and runtime path.

## Notes

The existing `test-copilot-skills-frontmatter` tests skills frontmatter but not the `engine.behaviors` import mechanism introduced as the replacement for built-in engines. Open questions: what minimal `engine.behaviors` overlay is valid for Copilot in the test environment?

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `awmgmcpg`
>> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "awmgmcpg"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/29403625307) · 30.2 AIC · ⌖ 5.72 AIC · ⊞ 5.6K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing test-copilot-skills-frontmatter workflow and the proposed test-copilot-custom-engine-behaviors.md file. Determine which minimal engine.behaviors frontmatter overlay is valid for Copilot, then run the workflow_dispatch test and confirm that the agent creates an issue showing the behavior was applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, markdown
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.