[Failing test]: AzureResourcePreparerTests.PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource — race in ValidateComputeEnvironmentBindings
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Build information
Build: https://github.com/microsoft/aspire/actions/runs/31555210276
Build error leg or test failing: `Tests / Hosting.Azure / Hosting.Azure (ubuntu-latest)` — `Aspire.Hosting.Azure.Tests.AzureResourcePreparerTests.PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource`
Observed on PR #19219 (job `93986712070`). 1 failed of 1669. The test is **not** quarantined and has no prior tracking issue.
### Fill in the error message template
## Error Message
```json
{
"ErrorMessage": "Step 'validate-compute-environments' failed: Collection was modified; enumeration operation may not execute.",
"ErrorPattern": "",
"BuildRetry": false,
"ExcludeConsoleLog": false
}
```
### Error details
```yml
Error Message:
System.InvalidOperationException : Step 'validate-compute-environments' failed: Collection was modified; enumeration operation may not execute.
---- System.InvalidOperationException : Collection was modified; enumeration operation may not execute.
Stack Trace:
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, PipelineStepContext stepContext) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 1148
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.<>c__DisplayClass24_0.<g__ExecuteStepWithDependencies|0>d.MoveNext() in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 913
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepsAsTaskDag(List`1 steps, Dictionary`2 stepsByName, PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 951
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 585
at Aspire.Hosting.Azure.Tests.AzureResourcePreparerTests.PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource() in /_/tests/Aspire.Hosting.Azure.Tests/AzureResourcePreparerTests.cs:line 280
----- Inner Stack Trace -----
at System.Collections.Generic.List`1.Enumerator.MoveNext()
at System.Linq.Enumerable.OfTypeIterator[TResult](IEnumerable source)+MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ValidateComputeEnvironmentBindings(DistributedApplicationModel model) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 391
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.<>c.<.ctor>b__4_16(PipelineStepContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 356
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, PipelineStepContext stepContext) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 1136
```
### Analysis
This is a genuine concurrency bug in **core hosting**, not in the test or in the Azure integration.
`ValidateComputeEnvironmentBindings` enumerates the live application model:
https://github.com/microsoft/aspire/blob/main/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs#L391
```csharp
var computeEnvironments = model.Resources.OfType().ToList();
```
`model.Resources` is a `List`. The `validate-compute-environments` step runs inside `ExecuteStepsAsTaskDag`, which executes independent steps **concurrently**. Another step that adds or removes resources during that window invalidates the enumerator mid-`ToList()`, producing `Collection was modified`.
Because it depends on step interleaving, it fails intermittently. The same CI leg failed on an earlier run of the same PR with a completely different cause (a SQL port collision), which is consistent with flakiness rather than a regression — the PR under test only touches `Aspire.Hosting.Azure.Kubernetes` and does not touch `DistributedApplicationPipeline.cs`.
The repo's own test-review guidance calls out exactly this shape under "Thread-unsafe collections": a non-thread-safe collection mutated from concurrent callers running via the task DAG.
### Suggested direction
Some candidates, in rough order of preference:
1. Snapshot `model.Resources` once under the model's synchronization before validation runs, and validate the snapshot.
2. Make the `validate-compute-environments` step ordered so it cannot overlap steps that mutate the model.
3. Make `DistributedApplicationModel.Resources` safe to enumerate concurrently.
Whichever is chosen, the fix belongs in `src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs` and should come with a regression test that reproduces the interleaving deterministically rather than relying on timing.
### Note
Since this reproduces only under contention, please confirm any fix with a repeated/stress run rather than a single green pass.
Contributor guide
Research direction
Start with src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs, especially ValidateComputeEnvironmentBindings and ExecuteStepsAsTaskDag, then read tests/Aspire.Hosting.Azure.Tests/AzureResourcePreparerTests.cs around PipelineStepAfterBeforeStartCanInspectRoleAssignmentsForTargetAzureResource. Reproduce the concurrent model access deterministically, implement a safe fix, and verify it with a regression test plus a repeated or stress run without collection-modification failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100