Intermittent ArgumentNullException (Parameter 'key') in AzureProvisioningResource.EnsureParametersAlign — data race in concurrent Bicep template generation
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- I have searched the existing issues (no issue references `EnsureParametersAlign` or concurrent Bicep template generation).
### Describe the bug
`aspire deploy`/`aspire do ... --list-steps` intermittently (~50% on my machine) crashes during the pipeline configuration phase while generating Bicep templates:
```
System.ArgumentNullException: Value cannot be null. (Parameter 'key')
at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
at System.Collections.Generic.Dictionary`2.ContainsKey(TKey key)
at Aspire.Hosting.Azure.AzureProvisioningResource.EnsureParametersAlign(AzureResourceInfrastructure infrastructure) in /_/src/Aspire.Hosting.Azure/AzureProvisioningResource.cs:line 241
at Aspire.Hosting.Azure.AzureProvisioningResource.GetBicepTemplateFile(String directory, Boolean deleteTemporaryFileOnDispose) in /_/src/Aspire.Hosting.Azure/AzureProvisioningResource.cs:line 85
at Aspire.Hosting.Azure.AzureProvisioningResource.GetBicepTemplateString() in /_/src/Aspire.Hosting.Azure/AzureProvisioningResource.cs:line 109
at Aspire.Hosting.Azure.AzureBicepResource.<>c__DisplayClass0_0.<.ctor>b__1(PipelineConfigurationContext context) in /_/src/Aspire.Hosting.Azure/AzureBicepResource.cs:line 66
at Aspire.Hosting.Pipelines.PipelineConfigurationAnnotation.<>c__DisplayClass4_0.<.ctor>b__0(PipelineConfigurationContext context) in /_/src/Aspire.Hosting/Pipelines/PipelineConfigurationAnnotation.cs:line 42
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteConfigurationCallbacksAsync(PipelineContext pipelineContext, List`1 allSteps) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 803
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ResolveStepsAsync(PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 633
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 576
at Aspire.Hosting.Publishing.PipelineExecutor.ExecutePipelineAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/PipelineExecutor.cs:line 107
at Aspire.Hosting.Publishing.PipelineExecutor.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/Publishing/PipelineExecutor.cs:line 56
```
On other runs it surfaces as a bare `NullReferenceException` at the same site. The model is not changed between a passing and a failing run — this is a data race, not a deterministic modelling error.
### Root cause (a data race in concurrent Bicep template generation)
`EnsureParametersAlign` crashes at `foreach (var parameter in base.Parameters) { if (!dictionary.ContainsKey(parameter.Key)) ... }` because `parameter.Key` is `null`. `base.Parameters` is `AzureBicepResource.Parameters`, a plain `Dictionary`. A `Dictionary` cannot hold a null key through its indexer — so the null key is **dictionary corruption from unsynchronized concurrent writes**, not a value the app model ever inserted.
The concurrency:
- The configuration phase runs `PipelineConfigurationAnnotation` callbacks, each of which calls `AzureBicepResource.GetBicepTemplateString()`.
- `GetBicepTemplateString()` lazily populates `_generatedBicep` with **no synchronization**. When the same resource's generation runs on two threads before `_generatedBicep` is set, **both** threads execute `GetBicepTemplateFile()` → `ConfigureInfrastructure(infra)`.
- `ConfigureInfrastructure` (via `AsProvisioningParameter(...)`) mutates the resource's shared `AzureBicepResource.Parameters` dictionary: `infrastructure.AspireResource.Parameters[parameterName] = ...`. Two threads writing the same `Dictionary` concurrently corrupts it, yielding an entry with a null key.
- `EnsureParametersAlign` then does `dictionary.ContainsKey(parameter.Key)` with `parameter.Key == null` → `ArgumentNullException (Parameter 'key')`.
I confirmed two configuration passes run **concurrently** by instrumenting a global `AddPipelineConfiguration` callback with an `Interlocked` in-flight gauge and the managed thread id:
```
ATLI_DIAG enter tid=10 inflight=1
ATLI_DIAG total-bicep-resources=124 tid=10 inflight=1
ATLI_DIAG enter tid=11 inflight=2 <-- second thread enters while the first is still running
ATLI_DIAG total-bicep-resources=124 tid=11 inflight=2
ATLI_DIAG done tid=10
ATLI_DIAG done tid=11
```
In the same instrumentation, generating **every** resource's Bicep **serially** (one call at a time) produced 0 corrupted dictionaries across hundreds of generations; the corruption only appears when two passes overlap. The app model has ~124 Azure Bicep resources (an ACA environment with many container apps + role assignments), which widens the race window.
### To Reproduce
1. An AppHost that publishes to Azure Container Apps with a moderately large model (dozens of container apps / role assignments — more resources widens the window).
2. Run `aspire deploy --list-steps` (or `aspire do --list-steps`) repeatedly. ~50% of runs crash as above; the rest succeed with no model change.
### Exceptions (if any)
`System.ArgumentNullException: Value cannot be null. (Parameter 'key')` at `AzureProvisioningResource.EnsureParametersAlign` (see stack above); occasionally a bare `NullReferenceException` at the same site.
### .NET Version info
Aspire 13.4.6 (`Aspire.Hosting.Azure` 13.4.6, `Azure.Provisioning` 1.5.0). .NET 10. macOS (Apple Silicon).
### Suggested fixes
- Make `AzureBicepResource.GetBicepTemplateString()` / `_generatedBicep` population thread-safe (e.g. `Lazy` / a lock), so a resource's `ConfigureInfrastructure` runs at most once even under concurrent generation; and/or
- Ensure `ResolveStepsAsync` / `ExecuteConfigurationCallbacksAsync` cannot run concurrently for the same model, or make `AzureBicepResource.Parameters` mutation during generation thread-safe.
Contributor guide
Research direction
Start with AzureProvisioningResource.EnsureParametersAlign and GetBicepTemplateFile in src/Aspire.Hosting.Azure/AzureProvisioningResource.cs, then trace AzureBicepResource.GetBicepTemplateString and the pipeline callbacks in AzureBicepResource.cs and DistributedApplicationPipeline.cs. Reproduce with aspire deploy --list-steps on a large Azure Container Apps model and verify concurrent generation no longer corrupts Parameters or intermittently throws.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- cloud, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100