microsoft / microsoft/aspire

Race condition in resource eventing - OnResourceStopped fires twice and breaks lifecycle handlers

Open
#17,745 4 comments 1 reaction 0 assignees View on GitHub
area-app-model triage:bot-seen
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

I have an action in a resource which runs `IApplicationLifetime.StopApplication` and should start again afterwards. So for integration with Aspire I added an `OnResourceStopped` event handler to restart the resource when this happens.

```csharp
builder.OnResourceStopped(
async (r, e, c) =>
{
if (e.ResourceEvent.Snapshot.ExitCode == 0)
{
var cmdSvc = e.Services.GetRequiredService();
await cmdSvc.ExecuteCommandAsync(r, "resource-start", c);
}
}
);
```
If I am issuing this `resource-start` command within the `OnResourceStopped` handler, I get two invocations of `OnResourceStopped` event handler, where the 2nd one gets canceled while I'm stepping through it in debug mode.

My problem is that now this is broken. The resource that has this annotation is stuck in _Waiting_ state, although all of its dependencies are now running, and it doesn't accept any commands after this flow - _Start Resource_ button is available, but clicking it just disables it and nothing else.

The resource, however, is running, it's just the object model of AppHost that doesn't see it.

Additionally, before 13.3 `OnResourceStopped` from the Aspire dashboard did not set the `Snapshot.ExitCode` - it was null, and I could distinguish the events - when resource exited on its own and when resource was requested to stop. Now there is no such distinction. I wish there was a property to get the stop reason.

### Expected Behavior

If I issue a command in an event handler it should be properly executed. Resource events should only fire once and not fire if another command is issued during event handling.

### Steps To Reproduce

Add the code above and try stopping the resource from the dashboard.

### Exceptions (if any)

If you start the resource from the dashboard after the 2nd stop command, this is what AppHost logs:
```
fail: Aspire.Hosting.Dcp.DcpExecutor[0]
Failed to start resource web
k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'GatewayTimeout', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Timeout: request did not complete within requested timeout - context deadline exceeded","reason":"Timeout","details":{},"code":504}

at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.ICustomObjectsOperations_CreateClusterCustomObjectWithHttpMessagesAsync[T](Object body, String group, String version, String plural, String dryRun, String fieldManager, String fieldValidation, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.CreateClusterCustomObjectWithHttpMessagesAsync(Object body, String group, String version, String plural, String dryRun, String fieldManager, String fieldValidation, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass17_0`1.<b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 137
--- End of stack trace from previous location ---
at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass28_0`1.<b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 483
--- End of stack trace from previous location ---
at Polly.ResiliencePipeline.<>c__10`1.<b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Aspire.Hosting.Dcp.KubernetesService.ExecuteWithRetry[TResult](DcpApiOperationType operationType, String resourceType, Func`2 operation, Func`2 isRetryable, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 480
at Aspire.Hosting.Dcp.DcpExecutor.<>c__DisplayClass45_1`1.<b__0>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/DcpExecutor.cs:line 590
--- End of stack trace from previous location ---
at Aspire.Hosting.Dcp.DcpExecutor.CreateDcpObjectsAsync[RT](IEnumerable`1 objects, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/DcpExecutor.cs:line 599
at Aspire.Hosting.Dcp.ExecutableCreator.CreateObjectAsync(RenderedModelResource`1 er, EmptyCreationContext context, ILogger resourceLogger, IDcpObjectFactory factory, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ExecutableCreator.cs:line 135
at Aspire.Hosting.Dcp.DcpExecutor.StartResourceAsync(IResourceReference resourceReference, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/DcpExecutor.cs:line 1025
Resource started: web
```

### Aspire doctor output

Aspire Environment Check
========================

.NET SDK
✅ .NET 10.0.300 installed (x64)

Container Runtime
⚠️ Docker client version 27.4.1 is below minimum required 28.0.0 ← active
Upgrade Docker to version 28.0.0 or later from: https://www.docker.com/products/docker-desktop
See: https://aka.ms/aspire/containers

Environment
✅ HTTPS development certificate is trusted

Summary: 2 passed, 1 warnings, 0 failed
For detailed prerequisites: https://aka.ms/aspire-prerequisites

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.