Allow limiting parallelism of aspire deploy/publish pipeline steps
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is your feature request related to a problem? Please describe.
`aspire deploy` (and `aspire publish`) execute all independent pipeline steps fully in parallel with no concurrency cap. For an AppHost with several compute resources requiring an image build, this means every `build-*` and `push-*` step for every resource starts at the same time — in our case 8 services, each with multiple image layers, pushed simultaneously to the same private registry.
Against a registry sitting behind a reverse proxy (Traefik, in our case) this routinely causes push failures under load:
```
error from registry: invalid content range
unexpected status from PUT request to https:///v2/.../blobs/uploads/...: 499 status code 499
failed to do request: Head "https:///v2/.../blobs/...": net/http: TLS handshake timeout
```
Retrying `aspire deploy` a few times eventually gets everything through (each retry a few more images already have their layers on the registry), but a single deploy essentially never succeeds end-to-end when every image push starts at once. Tuning proxy/registry timeouts helps, but a client-side way to bound how many pushes run concurrently would let us avoid overloading a registry that reasonably can't take 7-8 large concurrent uploads.
### Describe the solution you'd like
A supported way to cap how many independent pipeline steps run concurrently during `aspire deploy`/`aspire publish` — e.g.:
- A CLI flag, such as `aspire deploy --max-parallel-steps `, or
- A `PipelineOptions` setting configurable from the AppHost (e.g. `builder.Services.Configure(o => o.MaxDegreeOfParallelism = 2)`), or
- A per-resource/step opt-in to a named concurrency group (e.g. all `push-*` steps sharing a "registry-push" group capped at N concurrent).
Any of these would let us throttle registry pushes specifically without serializing the entire pipeline (build steps that don't hit the network could stay fully parallel).
### Additional context
I looked at the pipeline executor source (`src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs`) to see if this was already possible. `ExecuteAsync` always calls `ExecuteStepsAsTaskDag`, which starts every step with an unthrottled `Task.Run` — no semaphore, no `MaxDegreeOfParallelism`, nothing configurable that I could find. There's also an `ExecuteStepsSequentially` / `ExecuteStepSequentiallyAsync(stepName, ...)` path, but it's only used internally for running a single step (and its dependencies) via `aspire do ` — it isn't reachable for a full deploy/publish run.
As a workaround today, we pre-warm the registry by pushing each image individually with `docker push` before running `aspire deploy`, so most layers already exist by the time the deploy step re-pushes them — but this doesn't help on a first-time deploy of new images (e.g. after adding a new service to the AppHost).
### Environment
- Aspire CLI: 13.4.6+87fe259e4fc244c599019a7b1304c85a1488f248 (build 13.400.626.31906)
- AppHost: C# AppHost, `AddContainerRegistry` + `WithContainerRegistry` (`ASPIRECOMPUTE003` preview) targeting a private Docker registry
- OS: Windows 11
- .NET SDK: 10.0
Contributor guide
Research direction
Start in src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs, reading ExecuteAsync and ExecuteStepsAsTaskDag to understand how deploy and publish launch independent steps. Compare that path with ExecuteStepsSequentially and ExecuteStepSequentiallyAsync; done should provide a supported concurrency cap for full deploy/publish runs without requiring all steps to execute serially.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, docker
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100