ci(e2e): restore advisory PR E2E and scheduled main coverage
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem Statement
PR #8445 moved automatic live E2E from pull requests to pushes on `main`. It also removed the scheduled broad run and the required `E2E / PR Gate` controller.
That change prevented unreliable live E2E from blocking every merge. It also removed two feedback paths:
- automatic E2E against the latest PR commit before merge;
- recurring broad E2E that detects regressions outside changed-file selection.
The PR Review Advisor still recommends E2E jobs and targets from changed files. Nothing automatically executes or evaluates that recommendation. After merge, `Relevant E2E` independently rebuilds a plan from the merged diff.
Maintainers therefore cannot answer whether the Advisor recommendations would have detected recent regressions. A PR can also merge before any selected live E2E runs against its commit.
Restore the pre-merge signal without restoring the merge deadlock.
## Desired Behavior
### Automatic advisory PR E2E
After eligible deterministic PR CI completes, trusted workflow code should generate and validate an E2E plan for the latest PR commit. GitHub Actions should automatically run the authorized selection against that commit.
The result must remain advisory. A failure, timeout, skipped job, unavailable runner, or unavailable Advisor model must not become a required merge check.
The selection contract should:
- preserve the deterministic risk-plan floor when the Advisor model is unavailable;
- distinguish deterministic selections from model-added recommendations;
- validate every job and target against the trusted E2E inventory;
- bind the plan and result to the PR, source repository, latest PR commit, base commit, trusted workflow commit, and correlation ID;
- mark evidence from a superseded PR commit as not current;
- report why a recommended job or target did not run.
For example, if an internal PR changes a runtime path mapped to `mcp-bridge`, the PR should receive an advisory result for the authorized selection. A later PR commit must receive a new plan and result.
### Additional maintainer-selected coverage
A maintainer should be able to add authorized jobs or targets without editing `.github/workflows/e2e.yaml` in the candidate branch. The workflow must validate the requested selection and current PR identity before candidate checkout.
### Scheduled broad main coverage
GitHub Actions should start a documented broad E2E selection against `main` every three hours. This schedule must remain separate from release qualification and release-only staging validation.
Scheduled and push-triggered runs must not cancel or invalidate base-image publication or E2E evidence for another commit. Coordinate this behavior with #9541.
### Recommendation-quality evidence
Each PR plan should record:
- changed files;
- deterministic jobs and targets;
- model-added jobs and targets;
- executed jobs and targets;
- authorization or availability exclusions;
- outcome and duration for each execution.
Post-merge investigation should be able to determine whether a failed `Relevant E2E` job was selected for the originating PR. This evidence should support measurement of missed regressions, irrelevant selections, unavailable selections, and PR E2E duration.
## Constraints and Non-goals
- Do not make live E2E a required merge check.
- Do not run the full live suite on every PR.
- Do not restore the deleted PR E2E controller without reassessing its failure modes and complexity.
- Do not let model output authorize credential-bearing execution.
- Candidate code must not receive long-lived repository credentials unless a current repository maintainer explicitly authorizes that boundary.
- Fork PRs must remain credential-free unless a current repository maintainer reviews and authorizes the selected credential boundary.
- Do not add a broad workflow rerun or hide product, test, runner, or external-service failures.
- Do not change release-qualification judgment.
- Do not include individual E2E test repairs in this issue.
## Capability Slices
1. Automatically run the trusted deterministic selection for eligible PR revisions and publish a non-blocking result.
2. Add normalized Advisor recommendations, recommendation-quality evidence, and validated maintainer-selected expansion.
3. Add the three-hour broad `main` schedule with exact-commit concurrency and publication behavior.
Each slice should include its workflow tests, maintainer guidance, and observable evidence.
## Acceptance Criteria
- [ ] Eligible internal PR revisions automatically run the trusted selected E2E plan after deterministic PR CI.
- [ ] Advisor or model unavailability does not suppress the deterministic risk-plan floor.
- [ ] The PR result is advisory and is not a required merge context.
- [ ] A PR update produces a plan for the latest PR commit and does not present older evidence as current.
- [ ] Fork and credential-bearing selections enforce the documented maintainer authorization boundary.
- [ ] A maintainer can request additional authorized jobs or targets without changing candidate workflow code.
- [ ] The plan records deterministic selections, model additions, executed selections, exclusions, outcomes, and durations.
- [ ] Maintainers can compare a post-merge `Relevant E2E` failure with the originating PR plan.
- [ ] A documented broad E2E selection starts against `main` every three hours.
- [ ] Scheduled and push-triggered runs preserve exact-commit publication and evidence across concurrent merges and reruns.
- [ ] Deterministic workflow tests cover selection, authorization denial, model unavailability, superseded PR commits, execution failure, and concurrency.
- [ ] The implementation does not change release qualification or require new live E2E tests.
## Related Work
- #7451 implemented the former direct required PR E2E gate.
- #8445 removed automatic PR E2E and moved automatic selection to pushes on `main`.
- #7912 tracks E2E duration, cost, and selective scheduling.
- #9159 tracks E2E reliability and coverage confidence.
- #9541 tracks exact-commit base-image publication across concurrent runs.
## Category
Testing
## Checklist
- [x] I searched existing issues and this is not a duplicate.
- [x] I described the problem and desired behavior.
Contributor guide
Assessment
This issue has not been assessed yet.