githubnext / githubnext/ado-aw

test(smoke): report scheduled smoke failures as GitHub issues from the orchestrator

Open
#1,796 0 comments 0 reactions 0 assignees View on GitHub
test
Dominant language
Rust
Stars
23
Forks
8
Avg merge
4d 9h
Merged PRs (30d)
22

Description

## Context

PR #1791 removed the `smoke-failure-reporter` smoke case. It resolved its
targets by exact ADO **definition name**, which the lane model abolishes for
cases — a case is now a ref queued against a shared lane, not a definition —
and two of the three names it watched (`Daily safe-output smoke canary` 2545,
`Daily smoke az CLI access` 2546) are deleted at cutover. It could not be
repaired by editing names.

Its *intent* is still worth keeping:

> A failed **scheduled** smoke run should become a GitHub issue, because nobody
> watches ADO.

Right now nothing does this. A nightly candidate or daily released orchestrator
failure is visible only in ADO.

## Proposal

File the issue from the **orchestrator**, deterministically, with no agent in
the loop.

`scripts/ado-script/src/executor-e2e/github-issue.ts` (279 lines) already does
exactly this job for the executor-e2e harness: scoped PAT, dedupe by exact
title, structured body, and `diagnoseGitHubAuthFailure` for token problems. It
exports `fileFailureIssue`, `buildIssueTitle`, `renderIssueBody`,
`findOpenIssueByTitle` and `createGitHubIssue`, so the reusable surface is
already factored out.

Sketch:

1. Lift the generic parts of `executor-e2e/github-issue.ts` into a shared
module (or import it directly — it is test-harness code, not a shipped
bundle, so either is fine).
2. Add a failure step to `tests/smoke/orchestrator-steps.yml`, gated on
`failed()` **and** on the run being scheduled, so PR and manual runs stay
silent — their failures are already surfaced on the PR.
3. Title on the orchestrator + mode, e.g.
`[smoke-failure] ado-aw candidate compiler smoke (candidate)`, so dedupe
collapses a recurring nightly failure into one issue.
4. Body: the results table `renderResultsTable` already produces, plus the
build URL and the per-case ref names.

## Why this is better than what was deleted

| | old reporter | proposed |
| --- | --- | --- |
| Mechanism | `claude-sonnet-4.6` agent, 20 min | deterministic step, seconds |
| Decides "did it fail?" by | reading build logs via MCP | the orchestrator's own exit code |
| Credentials | GitHub PAT in an agent's reach | PAT in a non-agent step |
| Tooling | `azure-devops` MCP toolset | none |
| Target discovery | ADO definition names (now wrong) | the orchestrator itself |
| Coverage | 3 named definitions | every case, via the run that owns them |

## Credential note

No smoke case currently files GitHub issues, so `ADO_AW_GITHUB_TOKEN` is
provisioned nowhere and the `agentic` lane holds no GitHub PAT beyond
`GITHUB_TOKEN` (Copilot CLI auth). This work re-introduces an issues PAT — put
it on the **orchestrator** definitions, not the lane, so it stays outside every
agent's blast radius. `scripts/rotate-agentplayground-secrets.ps1` has a
comment marking where to add it.

## Acceptance

- [ ] A failed scheduled candidate run files exactly one issue
- [ ] A failed scheduled released run files exactly one issue
- [ ] A second consecutive failure updates/skips rather than filing a duplicate
- [ ] PR and manual runs file nothing
- [ ] A green run files nothing
- [ ] The PAT never appears in an Agent or Detection step

## References

- Removed in #1791 (`45cc6059`)
- Prior art: `scripts/ado-script/src/executor-e2e/github-issue.ts`
- `tests/smoke/README.md` — lanes and modes

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with scripts/ado-script/src/executor-e2e/github-issue.ts and tests/smoke/orchestrator-steps.yml, then read tests/smoke/README.md for orchestrator modes and lanes. Check scripts/rotate-agentplayground-secrets.ps1 for the credential placement note. Done means scheduled candidate and released failures file or deduplicate one issue, while PR, manual, and green runs remain silent and the PAT is outside Agent and Detection steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, typescript
Domain
ci-cd, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.