githubnext / githubnext/ado-aw
test(smoke): report scheduled smoke failures as GitHub issues from the orchestrator
- Dominant language
- Rust
- Stars
- 23
- Forks
- 8
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 22
Description
## Context
PR #1791 removed the `smoke-failure-reporter` smoke case. It resolved its
targets by exact ADO **definition name**, which the lane model abolishes for
cases — a case is now a ref queued against a shared lane, not a definition —
and two of the three names it watched (`Daily safe-output smoke canary` 2545,
`Daily smoke az CLI access` 2546) are deleted at cutover. It could not be
repaired by editing names.
Its *intent* is still worth keeping:
> A failed **scheduled** smoke run should become a GitHub issue, because nobody
> watches ADO.
Right now nothing does this. A nightly candidate or daily released orchestrator
failure is visible only in ADO.
## Proposal
File the issue from the **orchestrator**, deterministically, with no agent in
the loop.
`scripts/ado-script/src/executor-e2e/github-issue.ts` (279 lines) already does
exactly this job for the executor-e2e harness: scoped PAT, dedupe by exact
title, structured body, and `diagnoseGitHubAuthFailure` for token problems. It
exports `fileFailureIssue`, `buildIssueTitle`, `renderIssueBody`,
`findOpenIssueByTitle` and `createGitHubIssue`, so the reusable surface is
already factored out.
Sketch:
1. Lift the generic parts of `executor-e2e/github-issue.ts` into a shared
module (or import it directly — it is test-harness code, not a shipped
bundle, so either is fine).
2. Add a failure step to `tests/smoke/orchestrator-steps.yml`, gated on
`failed()` **and** on the run being scheduled, so PR and manual runs stay
silent — their failures are already surfaced on the PR.
3. Title on the orchestrator + mode, e.g.
`[smoke-failure] ado-aw candidate compiler smoke (candidate)`, so dedupe
collapses a recurring nightly failure into one issue.
4. Body: the results table `renderResultsTable` already produces, plus the
build URL and the per-case ref names.
## Why this is better than what was deleted
| | old reporter | proposed |
| --- | --- | --- |
| Mechanism | `claude-sonnet-4.6` agent, 20 min | deterministic step, seconds |
| Decides "did it fail?" by | reading build logs via MCP | the orchestrator's own exit code |
| Credentials | GitHub PAT in an agent's reach | PAT in a non-agent step |
| Tooling | `azure-devops` MCP toolset | none |
| Target discovery | ADO definition names (now wrong) | the orchestrator itself |
| Coverage | 3 named definitions | every case, via the run that owns them |
## Credential note
No smoke case currently files GitHub issues, so `ADO_AW_GITHUB_TOKEN` is
provisioned nowhere and the `agentic` lane holds no GitHub PAT beyond
`GITHUB_TOKEN` (Copilot CLI auth). This work re-introduces an issues PAT — put
it on the **orchestrator** definitions, not the lane, so it stays outside every
agent's blast radius. `scripts/rotate-agentplayground-secrets.ps1` has a
comment marking where to add it.
## Acceptance
- [ ] A failed scheduled candidate run files exactly one issue
- [ ] A failed scheduled released run files exactly one issue
- [ ] A second consecutive failure updates/skips rather than filing a duplicate
- [ ] PR and manual runs file nothing
- [ ] A green run files nothing
- [ ] The PAT never appears in an Agent or Detection step
## References
- Removed in #1791 (`45cc6059`)
- Prior art: `scripts/ado-script/src/executor-e2e/github-issue.ts`
- `tests/smoke/README.md` — lanes and modes
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with scripts/ado-script/src/executor-e2e/github-issue.ts and tests/smoke/orchestrator-steps.yml, then read tests/smoke/README.md for orchestrator modes and lanes. Check scripts/rotate-agentplayground-secrets.ps1 for the credential placement note. Done means scheduled candidate and released failures file or deduplicate one issue, while PR, manual, and green runs remain silent and the PAT is outside Agent and Detection steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, typescript
- Domain
- ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100