dotnet / dotnet/orleans

OrleansDashboard | Failed to address message Request GetRuntimeStatistics

Open
#9,850 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

Hi 👋

I have recently noticed a _Failed to address message Request_ error which can happen during shutdown on Silos with Orleans.Dashboard/OrleansDashboard enabled.

To be specific, the error seems to be caused by `CollectStatistics` which is executed by a timer on a 1second interval, and invokes a call to the `IManagementGrain` grain.

What I assume is that during shutdown it can happen that the `IManagementGrain` stops accepting new calls, but the timer can still fire since it was not disposed/stopped yet. This behavior then produces `Failed to address message Request [S10.1.10.70:11111:125054564 sys.svc.user.95B6CAE6/10.1.10.70:11111@125054564]->[ management/0] Orleans.Runtime.IManagementGrain.GetRuntimeStatistics(Orleans.Runtime.SiloAddress[]) #14F2E1C487C52AC4` exception/error.

After reviewing the source it seems expected, and I was hoping to see if that is true? And if so, is there a supported way to suppress those shutdown errors?

Code snippet of the code in question:
https://github.com/dotnet/orleans/blob/36c019316685f7110f54d15080dbccaebd3ce8db/src/Dashboard/Orleans.Dashboard/Implementation/SiloGrainService.cs#L43-L92

I'm aware this is more related to the `OrleansDashboard` library, but since the project was moved to this repo I decided to ask directly here.

## My Setup
I'm using:
* Orleans 9.2.1 and,
* OrleansDashboard 8.2.0.
with:
* Kubernetes Hosting and,
* AdoNet Clustering

## Steps to Reproduce
1. Start a silo with OrleansDashboard (`CounterUpdateIntervalMs` should be set to the default 1-second interval).
2. Trigger silo shutdown
3. Watch logs during shutdown: within 1 second, the dashboard timer fires and calls `IManagementGrain.GetRuntimeStatistics`, which now rejects requests resulting in the error.

## Full Exception
```
Messsage: Failed to address message Request [S10.1.10.70:11111:125054564 sys.svc.user.95B6CAE6/10.1.10.70:11111@125054564]->[ management/0] Orleans.Runtime.IManagementGrain.GetRuntimeStatistics(Orleans.Runtime.SiloAddress[]) #14F2E1C487C52AC4
Exception: Orleans.Runtime.OrleansMessageRejectionException: Message was queued for sending after outbound queue was stopped
at Orleans.Serialization.Invocation.ResponseCompletionSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 81
at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 117
at Orleans.Runtime.ActivityPropagationGrainCallFilter.Process(IGrainCallContext context, Activity activity) in /_/src/Orleans.Core/Diagnostics/ActivityPropagationGrainCallFilter.cs:line 51
at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 88
at Orleans.Runtime.GrainReferenceRuntime.InvokeMethodWithFiltersAsync[TResult](GrainReference reference, IInvokable request, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/GrainReferenceRuntime.cs:line 90
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 701
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 373
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 339
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.g__SendMessageAsync|40_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 476
```

Contributor guide

Open the contributing guide

Research direction

Start with src/Dashboard/Orleans.Dashboard/Implementation/SiloGrainService.cs around lines 43-92, especially CollectStatistics and its one-second timer. Reproduce the documented shutdown sequence and inspect whether the timer remains active after silo shutdown begins; done should establish the supported handling or suppression of the resulting GetRuntimeStatistics errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.