microsoft / microsoft/aspire

Intermittent ArgumentNullException (Parameter 'key') in AzureProvisioningResource.EnsureParametersAlign — data race in concurrent Bicep template generation

Open
#18,673 0 comments 0 reactions 0 assignees View on GitHub
area-integrations
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- I have searched the existing issues (no issue references `EnsureParametersAlign` or concurrent Bicep template generation).

### Describe the bug

`aspire deploy`/`aspire do ... --list-steps` intermittently (~50% on my machine) crashes during the pipeline configuration phase while generating Bicep templates:

```
System.ArgumentNullException: Value cannot be null. (Parameter 'key')
at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
at System.Collections.Generic.Dictionary`2.ContainsKey(TKey key)
at Aspire.Hosting.Azure.AzureProvisioningResource.EnsureParametersAlign(AzureResourceInfrastructure infrastructure) in /_/src/Aspire.Hosting.Azure/AzureProvisioningResource.cs:line 241
at Aspire.Hosting.Azure.AzureProvisioningResource.GetBicepTemplateFile(String directory, Boolean deleteTemporaryFileOnDispose) in /_/src/Aspire.Hosting.Azure/AzureProvisioningResource.cs:line 85
at Aspire.Hosting.Azure.AzureProvisioningResource.GetBicepTemplateString() in /_/src/Aspire.Hosting.Azure/AzureProvisioningResource.cs:line 109
at Aspire.Hosting.Azure.AzureBicepResource.<>c__DisplayClass0_0.<.ctor>b__1(PipelineConfigurationContext context) in /_/src/Aspire.Hosting.Azure/AzureBicepResource.cs:line 66
at Aspire.Hosting.Pipelines.PipelineConfigurationAnnotation.<>c__DisplayClass4_0.<.ctor>b__0(PipelineConfigurationContext context) in /_/src/Aspire.Hosting/Pipelines/PipelineConfigurationAnnotation.cs:line 42
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteConfigurationCallbacksAsync(PipelineContext pipelineContext, List`1 allSteps) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 803
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ResolveStepsAsync(PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 633
at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(PipelineContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 576
at Aspire.Hosting.Publishing.PipelineExecutor.ExecutePipelineAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/PipelineExecutor.cs:line 107
at Aspire.Hosting.Publishing.PipelineExecutor.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/Publishing/PipelineExecutor.cs:line 56
```

On other runs it surfaces as a bare `NullReferenceException` at the same site. The model is not changed between a passing and a failing run — this is a data race, not a deterministic modelling error.

### Root cause (a data race in concurrent Bicep template generation)

`EnsureParametersAlign` crashes at `foreach (var parameter in base.Parameters) { if (!dictionary.ContainsKey(parameter.Key)) ... }` because `parameter.Key` is `null`. `base.Parameters` is `AzureBicepResource.Parameters`, a plain `Dictionary`. A `Dictionary` cannot hold a null key through its indexer — so the null key is **dictionary corruption from unsynchronized concurrent writes**, not a value the app model ever inserted.

The concurrency:

- The configuration phase runs `PipelineConfigurationAnnotation` callbacks, each of which calls `AzureBicepResource.GetBicepTemplateString()`.
- `GetBicepTemplateString()` lazily populates `_generatedBicep` with **no synchronization**. When the same resource's generation runs on two threads before `_generatedBicep` is set, **both** threads execute `GetBicepTemplateFile()` → `ConfigureInfrastructure(infra)`.
- `ConfigureInfrastructure` (via `AsProvisioningParameter(...)`) mutates the resource's shared `AzureBicepResource.Parameters` dictionary: `infrastructure.AspireResource.Parameters[parameterName] = ...`. Two threads writing the same `Dictionary` concurrently corrupts it, yielding an entry with a null key.
- `EnsureParametersAlign` then does `dictionary.ContainsKey(parameter.Key)` with `parameter.Key == null` → `ArgumentNullException (Parameter 'key')`.

I confirmed two configuration passes run **concurrently** by instrumenting a global `AddPipelineConfiguration` callback with an `Interlocked` in-flight gauge and the managed thread id:

```
ATLI_DIAG enter tid=10 inflight=1
ATLI_DIAG total-bicep-resources=124 tid=10 inflight=1
ATLI_DIAG enter tid=11 inflight=2 <-- second thread enters while the first is still running
ATLI_DIAG total-bicep-resources=124 tid=11 inflight=2
ATLI_DIAG done tid=10
ATLI_DIAG done tid=11
```

In the same instrumentation, generating **every** resource's Bicep **serially** (one call at a time) produced 0 corrupted dictionaries across hundreds of generations; the corruption only appears when two passes overlap. The app model has ~124 Azure Bicep resources (an ACA environment with many container apps + role assignments), which widens the race window.

### To Reproduce

1. An AppHost that publishes to Azure Container Apps with a moderately large model (dozens of container apps / role assignments — more resources widens the window).
2. Run `aspire deploy --list-steps` (or `aspire do --list-steps`) repeatedly. ~50% of runs crash as above; the rest succeed with no model change.

### Exceptions (if any)

`System.ArgumentNullException: Value cannot be null. (Parameter 'key')` at `AzureProvisioningResource.EnsureParametersAlign` (see stack above); occasionally a bare `NullReferenceException` at the same site.

### .NET Version info

Aspire 13.4.6 (`Aspire.Hosting.Azure` 13.4.6, `Azure.Provisioning` 1.5.0). .NET 10. macOS (Apple Silicon).

### Suggested fixes

- Make `AzureBicepResource.GetBicepTemplateString()` / `_generatedBicep` population thread-safe (e.g. `Lazy` / a lock), so a resource's `ConfigureInfrastructure` runs at most once even under concurrent generation; and/or
- Ensure `ResolveStepsAsync` / `ExecuteConfigurationCallbacksAsync` cannot run concurrently for the same model, or make `AzureBicepResource.Parameters` mutation during generation thread-safe.

Contributor guide

Open the contributing guide

Research direction

Start with AzureProvisioningResource.EnsureParametersAlign and GetBicepTemplateFile in src/Aspire.Hosting.Azure/AzureProvisioningResource.cs, then trace AzureBicepResource.GetBicepTemplateString and the pipeline callbacks in AzureBicepResource.cs and DistributedApplicationPipeline.cs. Reproduce with aspire deploy --list-steps on a large Azure Container Apps model and verify concurrent generation no longer corrupts Parameters or intermittently throws.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
cloud, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.