githubnext / githubnext/ado-aw

test(executor-e2e): add create-github-issue and set-github-issue-type scenarios

Open
#1,798 0 comments 0 reactions 0 assignees View on GitHub
javascript test
Dominant language
Rust
Stars
23
Forks
8
Avg merge
4d 9h
Merged PRs (30d)
22

Description

# Task: add `create-github-issue` + `set-github-issue-type` scenarios to executor-e2e

Add deterministic Stage 3 end-to-end coverage for the two GitHub issue safe
outputs in `githubnext/ado-aw`. Both currently ship with **zero runtime
coverage** — their only proof is a wiremock unit test.

## Background

`create-github-issue` and `set-github-issue-type` were added by PR #1670
(merged, unreleased as of v0.48.0). That PR deliberately deferred E2E coverage
because the smoke suite was being reworked in parallel (PR #1791). The rework
has since landed and **removed the `smoke-failure-reporter` smoke case**, which
was the only thing exercising `create-github-issue` at runtime. Net effect:
coverage went from "thin" to "none".

`executor-e2e` excludes it by design today. `tests/executor-e2e/README.md`
says: _"Excluded (out of scope or GitHub-only): the GitHub-only
`create-issue`."_ That exclusion predates the tools being first-class public
safe outputs, and is what you are reversing.

**Confusable, do not be misled:** `scripts/ado-script/src/executor-e2e/github-issue.ts`
is the harness's *own failure reporter* (it files an issue when scenarios
fail). It is NOT a scenario under test. You may reuse its REST plumbing, but
do not mistake it for existing coverage.

## What to build

Add `scripts/ado-script/src/executor-e2e/scenarios/github-issue.ts` exporting
`githubIssueScenarios`, and register it in `scenarios/index.ts` (append to
`allScenarios` — the order is deterministic and existing entries must not move).

Cover, at minimum:

1. **`create-github-issue`** — files an issue, asserts it exists on GitHub with
the expected title (including the configured `title-prefix`), body, and
config-injected labels.
2. **`set-github-issue-type`** — sets the issue type on an existing issue.
3. **The same-run temporary-ID handoff** — this is the highest-value case and
the one with the weakest current proof. `create-github-issue` accepts an
optional `temporary_id`, and `set-github-issue-type` accepts an
`issue_number` that may be either a real number or that temporary id. A
single executor run must resolve the handoff. A wrong REST shape or a
scoping failure here is invisible today.

Consider also (use judgement, do not gold-plate):
- `allowed-labels` rejection — config is default-**deny** for labels, unlike
`set-github-issue-type.allowed` which is default-**allow**. Use
`expectedFailure` for the rejection path.
- Clearing an issue type: `issue_type: ""` is the documented clear operation.

## The contract you must follow

Read `scripts/ado-script/src/executor-e2e/scenario.ts` first — it is the
authoritative contract and is well commented. Key points:

- A scenario is `{ tool, config, setup, ndjson, assert, cleanup }` with an
optional `id` (**set `id` when several scenarios share one `tool`**),
`files`, `env`, and `expectedFailure`.
- `ndjson()` returns the executor entry **without** the `name` field — the
runner injects `name: `.
- No LLM is involved. You craft the NDJSON directly, run the real `ado-aw
execute` binary, then assert via REST.
- `ScenarioContext` gives you `buildId`, `prefix(tool)`
(`ado-aw-det--`), `workDir`, `log`, and `rest`.
- Throw `SkipError` from `setup()` when a precondition is missing, so an
incomplete environment records as *skipped*, not *failed*.
- **Cleanup gotcha:** if `setup()` throws a non-`SkipError`, `cleanup()` is
NOT called — tear down anything you partially created before rethrowing.
See `scenarios/pr.ts` `setupPr` for the established pattern.
- `assert()` may populate state that `cleanup()` needs (e.g. the created issue
number read off the executor result) — do that **before** any fallible check,
so a later assertion failure still leaves cleanup able to tear down.

`scenarios/wiki.ts` is the cleanest model: env-var override, REST discovery,
`SkipError`, deterministic naming, real cleanup.

## Two things that make this genuinely different from every existing scenario

**1. It asserts against GitHub, not ADO.** `ScenarioContext.rest` is `AdoRest`
and will not help you. Every existing scenario asserts via ADO REST. You need
GitHub REST for setup/assert/cleanup. Reuse the helpers in
`executor-e2e/github-issue.ts` (`findOpenIssueByTitle`, `createGitHubIssue`,
`diagnoseGitHubAuthFailure`) rather than writing a fourth GitHub client — lift
them into a shared module if that is cleaner, but do not duplicate them.

**2. GitHub issues cannot be deleted.** Every other scenario tears its objects
down completely. You cannot. `cleanup()` must **close** the issue (and prefer a
`[e2e]`-style title prefix plus the `ado-aw-det--…` marker so leaked
issues are identifiable and greppable). Call this limitation out explicitly in
a comment and in the README — a reviewer will otherwise reasonably ask why
cleanup is weaker here. Consider whether the target repo should be a dedicated
scratch repo rather than one carrying real issues.

## Environment

- Definition `2550` (executor e2e) **already has `EXECUTOR_E2E_GITHUB_TOKEN`
provisioned** — no new secret provisioning is required. Confirm the token's
scope actually permits issue-type mutation on the target repo; issue *types*
are an org-level construct and may need more than plain Issues write. If it
does not, `SkipError` is the correct response, not a hard failure.
- Existing env vars follow the `E2E_*` convention (`E2E_ISSUE_REPO`,
`E2E_WIKI_NAME`, `E2E_QUEUE_PIPELINE_ID`, …). Follow it, and treat an
unexpanded ADO macro (literal `$(NAME)`) as unset — `github-issue.ts` already
has that guard, reuse it.
- Issue types are a closed set defined by the repository/org owner. Discover
the available types via REST rather than hardcoding a name, and `SkipError`
when none exist — a hardcoded type will rot.

## Validation

```bash
cd scripts/ado-script
npm run typecheck
npx vitest run src/executor-e2e
npm run build:executor-e2e
```

Add unit coverage alongside the existing `src/executor-e2e/__tests__/`. Prove
the temporary-ID handoff with an assertion that genuinely fails if the
resolution breaks — mutation-check it by deliberately breaking the handoff and
confirming your test goes red. A test that passes whether or not the feature
works is worse than no test.

Then update `tests/executor-e2e/README.md`: remove `create-issue` from the
"Excluded (out of scope or GitHub-only)" line, document the new scenarios and
any new env vars, and state the close-not-delete cleanup limitation.

## Out of scope

- Do not add these tools to the smoke suite (`tests/smoke/cases.json`) — the
point of executor-e2e is deterministic Stage 3 coverage with no agent.
- Do not add a GitHub PAT to the smoke `agentic` lane. Issue #1796 covers
orchestrator-side failure reporting separately and puts the token on the
orchestrator, not the lane.
- Do not change the Rust implementation of either tool. If you find a genuine
bug, report it rather than fixing it inline.

## Definition of done

- Scenarios registered and passing against the real project
- Temporary-ID handoff proven, mutation-checked
- Cleanup closes every issue it creates, limitation documented
- Missing preconditions skip rather than fail
- README exclusion line corrected
- `npm run typecheck`, `npx vitest run src/executor-e2e`,
`npm run build:executor-e2e` all green

Contributor guide

No contributing guide indexed for this repository

Research direction

Read scripts/ado-script/src/executor-e2e/scenario.ts and use scenarios/wiki.ts as the model, then inspect executor-e2e/github-issue.ts and scenarios/index.ts. Run the listed typecheck, Vitest, and build commands while adding the GitHub scenarios, unit coverage, and README updates. Done means the real scenarios pass, temporary-ID resolution is mutation-checked, missing prerequisites skip, and created issues are closed during cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, node.js, typescript
Domain
devtools, documentation, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.