githubnext / githubnext/gh-aw-test

Suggested test: copilot/custom-engine-threat-detection

Open
#14,834 0 comments 0 reactions 0 assignees View on GitHub
suggested new test
Dominant language
Shell
Stars
8
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Link to the gh-aw PR: github/gh-aw#59636 — Fix threat detection config_error for workflows using custom engines

## Proposed test

- **Workflow file**: `test-copilot-custom-engine-threat-detection.md`
- **Trigger**: `workflow_dispatch`
- **Engine**: custom `engine.id` (non-built-in) with `detection-engine: copilot`
- **Safe output**: `create-issue`
- **Variant**: standard

## Minimal test prompt sketch

Define a workflow using a custom engine definition with `detection-engine: copilot`, plus a trivial `create-issue` safe output with a `samples:` block. Assert the compiled `.lock.yml` threat-detection step passes `--engine copilot` (not the custom ID), and that the run completes without `THREAT_DETECTION_STATUS: reason=config_error`.

## New fixtures or secrets needed

Needs a custom engine definition usable in this repo (no such fixture currently exists here) — flagging for triage on whether a minimal custom engine can be added safely.

## Notes

No existing test uses a custom `engine.id` or `detection-engine` key. This is compiler/threat-detection plumbing, verifiable from the compiled lockfile with `--use-samples`, so no live custom-engine credentials should be required.

> Generated by [🔍 Suggest New E2E Tests](https://github.com/githubnext/gh-aw-test/actions/runs/34456199311) · copilot · auto · 54.6 AIC · ⌖ 6.16 AIC · ⊞ 8.4K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-test+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-test%2Fsuggest-new-e2e-tests%22&type=issues)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the proposed test-copilot-custom-engine-threat-detection.md workflow and review how its compiled .lock.yml output represents the threat-detection step. Determine whether a minimal custom engine fixture can be added safely, then verify with --use-samples that the step uses --engine copilot and avoids the config_error status.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, security, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.