[automated] Track conditional PR CI routing defects found by live audit
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
[automated]
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
A live exact-head audit of conditional PR CI found one confirmed safety defect,
three systematic over-selection defects, and one unconditional job that should
participate in selection. The audit replayed the selector and
`eng/github-ci/test-trigger-map.yml` from each PR head, then independently
derived the expected projects/jobs from ProjectGraph, runtime dependencies,
workflow consumers, and loose inputs.
Across 228 enforcing-mode decisions audited so far, 2 runs had the same
false-negative defect, 25 had confirmed false-positive jobs, and 4 rename runs
safely but unnecessarily selected `ALL`. The cases below are distinct defects
but share the same selector/map ownership and should be tracked together.
#### 1. Runtime Hosting-package consumers can be skipped
PR #19720 changed publish/deploy behavior and several Hosting integrations. Its
selector chose 49 of 99 PR test projects but omitted
`Aspire.Cli.EndToEnd.Tests`, so `Tests / Build CLI E2E Docker image` was skipped
in runs
[33595798642](https://github.com/microsoft/aspire/actions/runs/33595798642)
and
[33458260779](https://github.com/microsoft/aspire/actions/runs/33458260779).
`Aspire.Cli.EndToEnd.Tests` exercises the affected `aspire publish` and
`aspire deploy` paths using built NuGet packages, but those runtime package
dependencies are not visible through its ordinary `ProjectReference` graph and
are not represented in Layer 2. This is a confirmed false negative.
#### 2. Broad Hosting rules run extension E2E for unrelated integrations
The current Hosting path/project rules select the full VS Code extension E2E
workflow for leaf integrations that no extension fixture restores, loads, or
references. Confirmed examples include PRs #17742, #19024, #19243, #19810, and
#19826.
Across 24 audited runs, this instantiated 772 unnecessary extension E2E jobs
and consumed about 3,388 aggregate runner-minutes, excluding extension
bootstrap prerequisites. For example, PR #19810's Azure Sandbox-only diff
added about 180 runner-minutes in
[run 33596440076](https://github.com/microsoft/aspire/actions/runs/33596440076).
#### 3. CLI test-only changes run extension E2E
PR #19727 changed one file in `Aspire.Cli.Tests` and one in
`Aspire.Cli.EndToEnd.Tests`. The selector correctly selected those .NET test
projects, but the two broad test-directory rules also instantiated 35
extension E2E jobs consuming about 163 aggregate runner-minutes in
[run 33432144442](https://github.com/microsoft/aspire/actions/runs/33432144442).
The extension workflow consumes production CLI/package artifacts; it does not
build or run either changed test project.
#### 4. Rename tombstones force `ALL`
PR #19486 renamed:
```text
eng/scripts/aspire-skills-bundle.common.ps1
-> eng/scripts/aspire-skills-bundles.common.ps1
```
The exact-head map routes the new path to `Infrastructure.Tests`, but the
selector also evaluates the deleted old path under `--no-renames`. That path
cannot match the map at `HEAD`, so
[run 33414997820](https://github.com/microsoft/aspire/actions/runs/33414997820)
escalated to `ALL`, selecting 99 unrelated test projects and four unrelated
gated jobs. The same behavior occurred in three subsequent runs of the PR.
This fallback is safe but unnecessarily expensive when the rename destination
is already attributed.
#### 5. Stabilization Check is unconditional
`stabilization_check` lives outside the reusable `tests.yml` workflow and does
not consume selector output. PR #19455 changed only `analyze-ci-failure`
workflow files and an `Infrastructure.Tests` file; the selector correctly chose
only `Infrastructure.Tests` and no gated jobs, but Stabilization Check still
ran in
[run 33678547895](https://github.com/microsoft/aspire/actions/runs/33678547895).
Six measured runs averaged 6m39s on `8-core-ubuntu-latest`. The job's useful
coverage is stable-package metadata/dependency validation, eleven
version-sensitive CLI test classes, and the stable-feed init/restore smoke
test—not arbitrary source, workflow, or infrastructure-test changes.
#### Unresolved routing questions
These are not classified as defects yet because independent reviews disagreed
materially:
- PR #19854 /
[run 33582827098](https://github.com/microsoft/aspire/actions/runs/33582827098):
whether a TypeScript production module with precise unit coverage but no E2E
assertion should select extension E2E.
- PR #19673 /
[run 33689651134](https://github.com/microsoft/aspire/actions/runs/33689651134):
whether generic CLI artifact/runtime reachability makes installer and API
jobs relevant.
- PR #19878 /
[run 33687040476](https://github.com/microsoft/aspire/actions/runs/33687040476):
whether eager scanner-constructor resolution makes every CLI-consuming job
relevant, or only jobs that reach scanner behavior.
### Expected Behavior
Conditional PR CI should have zero high-confidence false negatives while
selecting only test projects and gated jobs that can consume the changed
production code, test source, workflow input, or runtime package.
Specifically:
- Runtime NuGet consumers such as CLI E2E must be represented when ProjectGraph
cannot see them.
- Extension E2E must be routed by actual production/fixture dependencies rather
than every Hosting integration or CLI test file.
- A mapped rename should not force `ALL` solely because the deleted source path
is absent from the head map.
- Stabilization Check should participate in the same auditable selection
decision, while preserving fail-safe `ALL` behavior.
- The three unresolved cases should receive explicit routing policy decisions
and focused acceptance coverage.
### Steps To Reproduce
1. Check out PR #19720 head `954fb57812f08b31cc868ead7ba7f7b87fd750bc`
with base `8e47a3dbf220a74a79fd734ea7b4b941c842595e` available.
2. Run the exact enforcing selector:
```bash
SELECT_TESTS_COMMENT_FILE=/tmp/select-tests-comment.md \
SELECT_TESTS_JSON_FILE=/tmp/select-tests-selection.json \
./dotnet.sh run --project tools/SelectTests/SelectTests.csproj -- \
--repo-root . \
--before-build-props /tmp/BeforeBuildProps.props \
--map eng/github-ci/test-trigger-map.yml \
--enforce \
--from 8e47a3dbf220a74a79fd734ea7b4b941c842595e \
--to 954fb57812f08b31cc868ead7ba7f7b87fd750bc
```
3. Inspect `/tmp/select-tests-comment.md`. `Aspire.Cli.EndToEnd.Tests` is
absent despite exercising the affected publish/deploy packages at runtime.
4. Use the linked PRs and workflow runs above for the over-selection, rename,
and unconditional Stabilization Check examples.
### Exceptions (if any)
N/A
### Aspire doctor output
N/A - this affects GitHub Actions routing.
### Anything else?
Relevant implementation and configuration:
- `tools/SelectTests/`
- `eng/github-ci/test-trigger-map.yml`
- `.github/workflows/tests.yml`
- `.github/workflows/ci.yml`
The extension false positives and unconditional Stabilization Check are
inherited routing behavior rather than selector-computation regressions. False
positives reduce efficiency; the skipped CLI E2E coverage is the
safety-critical finding. No existing issue or pull request matching these
routing defects was found in targeted searches.
Contributor guide
Research direction
Start with tools/SelectTests/, eng/github-ci/test-trigger-map.yml, .github/workflows/tests.yml, and .github/workflows/ci.yml. Reproduce the PR #19720 selector command and inspect the generated selection output, then compare it with the documented audit cases. Done means runtime consumers are represented, unrelated extension jobs are excluded, mapped renames avoid unnecessary ALL fallback, Stabilization Check is selectable, and unresolved routing cases have explicit acceptance coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions
- Domain
- ci-cd, tooling
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100