githubnext / githubnext/ado-aw
test(executor-e2e): add create-github-issue and set-github-issue-type scenarios
- Dominant language
- Rust
- Stars
- 23
- Forks
- 8
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 22
Description
# Task: add `create-github-issue` + `set-github-issue-type` scenarios to executor-e2e
Add deterministic Stage 3 end-to-end coverage for the two GitHub issue safe
outputs in `githubnext/ado-aw`. Both currently ship with **zero runtime
coverage** — their only proof is a wiremock unit test.
## Background
`create-github-issue` and `set-github-issue-type` were added by PR #1670
(merged, unreleased as of v0.48.0). That PR deliberately deferred E2E coverage
because the smoke suite was being reworked in parallel (PR #1791). The rework
has since landed and **removed the `smoke-failure-reporter` smoke case**, which
was the only thing exercising `create-github-issue` at runtime. Net effect:
coverage went from "thin" to "none".
`executor-e2e` excludes it by design today. `tests/executor-e2e/README.md`
says: _"Excluded (out of scope or GitHub-only): the GitHub-only
`create-issue`."_ That exclusion predates the tools being first-class public
safe outputs, and is what you are reversing.
**Confusable, do not be misled:** `scripts/ado-script/src/executor-e2e/github-issue.ts`
is the harness's *own failure reporter* (it files an issue when scenarios
fail). It is NOT a scenario under test. You may reuse its REST plumbing, but
do not mistake it for existing coverage.
## What to build
Add `scripts/ado-script/src/executor-e2e/scenarios/github-issue.ts` exporting
`githubIssueScenarios`, and register it in `scenarios/index.ts` (append to
`allScenarios` — the order is deterministic and existing entries must not move).
Cover, at minimum:
1. **`create-github-issue`** — files an issue, asserts it exists on GitHub with
the expected title (including the configured `title-prefix`), body, and
config-injected labels.
2. **`set-github-issue-type`** — sets the issue type on an existing issue.
3. **The same-run temporary-ID handoff** — this is the highest-value case and
the one with the weakest current proof. `create-github-issue` accepts an
optional `temporary_id`, and `set-github-issue-type` accepts an
`issue_number` that may be either a real number or that temporary id. A
single executor run must resolve the handoff. A wrong REST shape or a
scoping failure here is invisible today.
Consider also (use judgement, do not gold-plate):
- `allowed-labels` rejection — config is default-**deny** for labels, unlike
`set-github-issue-type.allowed` which is default-**allow**. Use
`expectedFailure` for the rejection path.
- Clearing an issue type: `issue_type: ""` is the documented clear operation.
## The contract you must follow
Read `scripts/ado-script/src/executor-e2e/scenario.ts` first — it is the
authoritative contract and is well commented. Key points:
- A scenario is `{ tool, config, setup, ndjson, assert, cleanup }` with an
optional `id` (**set `id` when several scenarios share one `tool`**),
`files`, `env`, and `expectedFailure`.
- `ndjson()` returns the executor entry **without** the `name` field — the
runner injects `name: `.
- No LLM is involved. You craft the NDJSON directly, run the real `ado-aw
execute` binary, then assert via REST.
- `ScenarioContext` gives you `buildId`, `prefix(tool)`
(`ado-aw-det--`), `workDir`, `log`, and `rest`.
- Throw `SkipError` from `setup()` when a precondition is missing, so an
incomplete environment records as *skipped*, not *failed*.
- **Cleanup gotcha:** if `setup()` throws a non-`SkipError`, `cleanup()` is
NOT called — tear down anything you partially created before rethrowing.
See `scenarios/pr.ts` `setupPr` for the established pattern.
- `assert()` may populate state that `cleanup()` needs (e.g. the created issue
number read off the executor result) — do that **before** any fallible check,
so a later assertion failure still leaves cleanup able to tear down.
`scenarios/wiki.ts` is the cleanest model: env-var override, REST discovery,
`SkipError`, deterministic naming, real cleanup.
## Two things that make this genuinely different from every existing scenario
**1. It asserts against GitHub, not ADO.** `ScenarioContext.rest` is `AdoRest`
and will not help you. Every existing scenario asserts via ADO REST. You need
GitHub REST for setup/assert/cleanup. Reuse the helpers in
`executor-e2e/github-issue.ts` (`findOpenIssueByTitle`, `createGitHubIssue`,
`diagnoseGitHubAuthFailure`) rather than writing a fourth GitHub client — lift
them into a shared module if that is cleaner, but do not duplicate them.
**2. GitHub issues cannot be deleted.** Every other scenario tears its objects
down completely. You cannot. `cleanup()` must **close** the issue (and prefer a
`[e2e]`-style title prefix plus the `ado-aw-det--…` marker so leaked
issues are identifiable and greppable). Call this limitation out explicitly in
a comment and in the README — a reviewer will otherwise reasonably ask why
cleanup is weaker here. Consider whether the target repo should be a dedicated
scratch repo rather than one carrying real issues.
## Environment
- Definition `2550` (executor e2e) **already has `EXECUTOR_E2E_GITHUB_TOKEN`
provisioned** — no new secret provisioning is required. Confirm the token's
scope actually permits issue-type mutation on the target repo; issue *types*
are an org-level construct and may need more than plain Issues write. If it
does not, `SkipError` is the correct response, not a hard failure.
- Existing env vars follow the `E2E_*` convention (`E2E_ISSUE_REPO`,
`E2E_WIKI_NAME`, `E2E_QUEUE_PIPELINE_ID`, …). Follow it, and treat an
unexpanded ADO macro (literal `$(NAME)`) as unset — `github-issue.ts` already
has that guard, reuse it.
- Issue types are a closed set defined by the repository/org owner. Discover
the available types via REST rather than hardcoding a name, and `SkipError`
when none exist — a hardcoded type will rot.
## Validation
```bash
cd scripts/ado-script
npm run typecheck
npx vitest run src/executor-e2e
npm run build:executor-e2e
```
Add unit coverage alongside the existing `src/executor-e2e/__tests__/`. Prove
the temporary-ID handoff with an assertion that genuinely fails if the
resolution breaks — mutation-check it by deliberately breaking the handoff and
confirming your test goes red. A test that passes whether or not the feature
works is worse than no test.
Then update `tests/executor-e2e/README.md`: remove `create-issue` from the
"Excluded (out of scope or GitHub-only)" line, document the new scenarios and
any new env vars, and state the close-not-delete cleanup limitation.
## Out of scope
- Do not add these tools to the smoke suite (`tests/smoke/cases.json`) — the
point of executor-e2e is deterministic Stage 3 coverage with no agent.
- Do not add a GitHub PAT to the smoke `agentic` lane. Issue #1796 covers
orchestrator-side failure reporting separately and puts the token on the
orchestrator, not the lane.
- Do not change the Rust implementation of either tool. If you find a genuine
bug, report it rather than fixing it inline.
## Definition of done
- Scenarios registered and passing against the real project
- Temporary-ID handoff proven, mutation-checked
- Cleanup closes every issue it creates, limitation documented
- Missing preconditions skip rather than fail
- README exclusion line corrected
- `npm run typecheck`, `npx vitest run src/executor-e2e`,
`npm run build:executor-e2e` all green
Contributor guide
No contributing guide indexed for this repository
Research direction
Read scripts/ado-script/src/executor-e2e/scenario.ts and use scenarios/wiki.ts as the model, then inspect executor-e2e/github-issue.ts and scenarios/index.ts. Run the listed typecheck, Vitest, and build commands while adding the GitHub scenarios, unit coverage, and README updates. Done means the real scenarios pass, temporary-ID resolution is mutation-checked, missing prerequisites skip, and created issues are closed during cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, node.js, typescript
- Domain
- devtools, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100