microsoft / microsoft/aspire

[Failing test]: AzureResourcePreparerTests.PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource — race in ValidateComputeEnvironmentBindings

Open
#19,266 1 comment 0 reactions 0 assignees View on GitHub
area-app-model area-deployment area-engineering-systems failing-test flaky-test triage:bot-seen
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Build information

Build: https://github.com/microsoft/aspire/actions/runs/31555210276
Build error leg or test failing: `Tests / Hosting.Azure / Hosting.Azure (ubuntu-latest)` — `Aspire.Hosting.Azure.Tests.AzureResourcePreparerTests.PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource`

Observed on PR #19219 (job `93986712070`). 1 failed of 1669. The test is **not** quarantined and has no prior tracking issue.

### Fill in the error message template

## Error Message

```json
{
"ErrorMessage": "Step 'validate-compute-environments' failed: Collection was modified; enumeration operation may not execute.",
"ErrorPattern": "",
"BuildRetry": false,
"ExcludeConsoleLog": false
}
```

### Error details

```yml
Error Message:
System.InvalidOperationException : Step 'validate-compute-environments' failed: Collection was modified; enumeration operation may not execute.
---- System.InvalidOperationException : Collection was modified; enumeration operation may not execute.

Stack Trace:
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, PipelineStepContext stepContext) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 1148
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.<>c__DisplayClass24_0.<g__ExecuteStepWithDependencies|0>d.MoveNext() in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 913
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepsAsTaskDag(List`1 steps, Dictionary`2 stepsByName, PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 951
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 585
at Aspire.Hosting.Azure.Tests.AzureResourcePreparerTests.PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource() in /_/tests/Aspire.Hosting.Azure.Tests/AzureResourcePreparerTests.cs:line 280

----- Inner Stack Trace -----
at System.Collections.Generic.List`1.Enumerator.MoveNext()
at System.Linq.Enumerable.OfTypeIterator[TResult](IEnumerable source)+MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ValidateComputeEnvironmentBindings(DistributedApplicationModel model) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 391
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.<>c.<.ctor>b__4_16(PipelineStepContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 356
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, PipelineStepContext stepContext) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 1136
```

### Analysis

This is a genuine concurrency bug in **core hosting**, not in the test or in the Azure integration.

`ValidateComputeEnvironmentBindings` enumerates the live application model:

https://github.com/microsoft/aspire/blob/main/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs#L391

```csharp
var computeEnvironments = model.Resources.OfType().ToList();
```

`model.Resources` is a `List`. The `validate-compute-environments` step runs inside `ExecuteStepsAsTaskDag`, which executes independent steps **concurrently**. Another step that adds or removes resources during that window invalidates the enumerator mid-`ToList()`, producing `Collection was modified`.

Because it depends on step interleaving, it fails intermittently. The same CI leg failed on an earlier run of the same PR with a completely different cause (a SQL port collision), which is consistent with flakiness rather than a regression — the PR under test only touches `Aspire.Hosting.Azure.Kubernetes` and does not touch `DistributedApplicationPipeline.cs`.

The repo's own test-review guidance calls out exactly this shape under "Thread-unsafe collections": a non-thread-safe collection mutated from concurrent callers running via the task DAG.

### Suggested direction

Some candidates, in rough order of preference:

1. Snapshot `model.Resources` once under the model's synchronization before validation runs, and validate the snapshot.
2. Make the `validate-compute-environments` step ordered so it cannot overlap steps that mutate the model.
3. Make `DistributedApplicationModel.Resources` safe to enumerate concurrently.

Whichever is chosen, the fix belongs in `src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs` and should come with a regression test that reproduces the interleaving deterministically rather than relying on timing.

### Note

Since this reproduces only under contention, please confirm any fix with a repeated/stress run rather than a single green pass.

Contributor guide

Open the contributing guide

Research direction

Start with src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs, especially ValidateComputeEnvironmentBindings and ExecuteStepsAsTaskDag, then read tests/Aspire.Hosting.Azure.Tests/AzureResourcePreparerTests.cs around PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource. Reproduce the concurrent model access deterministically, implement a safe fix, and verify it with a regression test plus a repeated or stress run without collection-modification failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.