Add outerloop E2E tests that actually build and run PublishAsDockerFile output
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Is your feature request related to a problem? Please describe the problem.
The integrations that generate a Dockerfile via `PublishAsDockerFile` are covered by Verify snapshot tests only. Snapshots assert that we emit the text we intended to emit - they say nothing about whether that text produces an image that builds, or a container that starts.
This leaves two classes of failure completely uncovered:
1. **The generated Dockerfile is wrong.** The snapshot is green because the output matches what we decided to write; the build or the container fails anyway.
2. **An upstream image changes underneath us.** Nothing in the repo changes, so no test re-runs, and no snapshot moves. The first person to find out is a user running `aspire publish`.
The second is not hypothetical. These generators reference floating or periodically bumped base images (`rust:alpine`, `alpine:3.24`, and equivalents in the other integrations), so upstream is free to change the toolchain, the shipped packages or the user database at any time.
Recent concrete examples, all of which shipped green snapshots and were only caught by building the image by hand:
- `rust:1.89-alpine` has `gcc` but no libc headers (`musl-dev` only landed in the official Alpine Rust images in 1.92.0), so any crate needing a C dependency - `ring`, `openssl-sys` - failed to compile.
- `addgroup -g 999 -S app` failed because Alpine already uses gid 999 for `ping`, and the `groupadd` fallback does not exist on Alpine so it could not rescue it. The runtime stage never built.
- A `--mount=type=cache` over `RUSTUP_HOME` broke the build entirely: cache mounts start empty and shadow what they cover, hiding the toolchains the image ships (`rustup could not choose a version of cargo to run`).
- Cache-mounting the cargo target directory naively means `COPY --from=build` finds nothing, because cache mounts are not part of the resulting layer.
- `rustup target add` emitted before `COPY . .` installs the target into the image default toolchain, so a crate pinning a toolchain in `rust-toolchain.toml` builds without it.
Every one of these is invisible to a snapshot test and obvious within seconds of a real `docker build`.
### Describe the solution you would like
Add outerloop tests that take a playground app through the full publish path:
1. Publish the AppHost to a temp directory.
2. `docker build` the generated Dockerfile against the app build context.
3. `docker run` the resulting image.
4. Assert the container reaches a healthy state - hit a health endpoint, and check anything else we claim about the image (for example that it runs as the non-root `app` user rather than root).
Outerloop is the right home for these: they need a container runtime, they pull real base images, and they take minutes rather than seconds. `tests-outerloop.yml` already runs every 6 hours, which is also what gives us the upstream-drift coverage - a scheduled run fails when a base image changes even though no PR touched the repo.
Worth covering the integrations that generate Dockerfiles today:
- `Aspire.Hosting.Go`
- `Aspire.Hosting.JavaScript`
- `Aspire.Hosting.Python`
- `Aspire.Hosting.Rust`
- `Aspire.Hosting.Blazor` (gateway)
- the core `PublishAsDockerFile` paths in `ExecutableResourceBuilderExtensions` and `ProjectResourceBuilderExtensions`
A shared helper doing publish -> build -> run -> probe would make each integration test small, since the shape is identical across all of them.
### Additional context
This came out of review on #18906 (Rust hosting integration). The gap is not Rust-specific - Rust is just where it happened to be noticed, and the same snapshot-only coverage applies to the other app integrations, so it is being raised separately rather than fixed in that PR.
Contributor guide
Research direction
Start with the existing outerloop setup in tests-outerloop.yml and the PublishAsDockerFile paths in ExecutableResourceBuilderExtensions and ProjectResourceBuilderExtensions. Review the snapshot-only coverage for the listed Aspire.Hosting integrations, then design a shared publish-build-run-probe helper. Done means generated images build, containers start, health checks pass, and claimed non-root execution is verified on the scheduled outerloop run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, docker, go, java, javascript, python, rust
- Domain
- backend, devops, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100