OrleansDashboard | Failed to address message Request GetRuntimeStatistics
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
Hi 👋
I have recently noticed a _Failed to address message Request_ error which can happen during shutdown on Silos with Orleans.Dashboard/OrleansDashboard enabled.
To be specific, the error seems to be caused by `CollectStatistics` which is executed by a timer on a 1second interval, and invokes a call to the `IManagementGrain` grain.
What I assume is that during shutdown it can happen that the `IManagementGrain` stops accepting new calls, but the timer can still fire since it was not disposed/stopped yet. This behavior then produces `Failed to address message Request [S10.1.10.70:11111:125054564 sys.svc.user.95B6CAE6/10.1.10.70:11111@125054564]->[ management/0] Orleans.Runtime.IManagementGrain.GetRuntimeStatistics(Orleans.Runtime.SiloAddress[]) #14F2E1C487C52AC4` exception/error.
After reviewing the source it seems expected, and I was hoping to see if that is true? And if so, is there a supported way to suppress those shutdown errors?
Code snippet of the code in question:
https://github.com/dotnet/orleans/blob/36c019316685f7110f54d15080dbccaebd3ce8db/src/Dashboard/Orleans.Dashboard/Implementation/SiloGrainService.cs#L43-L92
I'm aware this is more related to the `OrleansDashboard` library, but since the project was moved to this repo I decided to ask directly here.
## My Setup
I'm using:
* Orleans 9.2.1 and,
* OrleansDashboard 8.2.0.
with:
* Kubernetes Hosting and,
* AdoNet Clustering
## Steps to Reproduce
1. Start a silo with OrleansDashboard (`CounterUpdateIntervalMs` should be set to the default 1-second interval).
2. Trigger silo shutdown
3. Watch logs during shutdown: within 1 second, the dashboard timer fires and calls `IManagementGrain.GetRuntimeStatistics`, which now rejects requests resulting in the error.
## Full Exception
```
Messsage: Failed to address message Request [S10.1.10.70:11111:125054564 sys.svc.user.95B6CAE6/10.1.10.70:11111@125054564]->[ management/0] Orleans.Runtime.IManagementGrain.GetRuntimeStatistics(Orleans.Runtime.SiloAddress[]) #14F2E1C487C52AC4
Exception: Orleans.Runtime.OrleansMessageRejectionException: Message was queued for sending after outbound queue was stopped
at Orleans.Serialization.Invocation.ResponseCompletionSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 81
at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 117
at Orleans.Runtime.ActivityPropagationGrainCallFilter.Process(IGrainCallContext context, Activity activity) in /_/src/Orleans.Core/Diagnostics/ActivityPropagationGrainCallFilter.cs:line 51
at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 88
at Orleans.Runtime.GrainReferenceRuntime.InvokeMethodWithFiltersAsync[TResult](GrainReference reference, IInvokable request, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/GrainReferenceRuntime.cs:line 90
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 701
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 373
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 339
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.g__SendMessageAsync|40_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 476
```
Contributor guide
Research direction
Start with src/Dashboard/Orleans.Dashboard/Implementation/SiloGrainService.cs around lines 43-92, especially CollectStatistics and its one-second timer. Reproduce the documented shutdown sequence and inspect whether the timer remains active after silo shutdown begins; done should establish the supported handling or suppression of the resulting GetRuntimeStatistics errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100