dotnet / dotnet/orleans

Unable to activate Grains in Orleans Runtime

Open
#8,774 1 comment 1 reaction 1 assignee Claimed by @ReubenBond View on GitHub
Needs: triage :mag:
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
15h 1m
Merged PRs (30d)
345

Description

I am facing issues in activating grains. I am successfully able to up the silos but the grain activation is throwing me TaskCanceledException.

Below are the logs might help understanding the problem:

```
[Error] Orleans.Grain: Error activating grain [Activation: S192.168.1.33:10025:58975009/Placement=RandomPlacement State=Activating]
System.Threading.Tasks.TaskCanceledException: Timed out waiting for OnActivateAsync to complete
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable(Task task, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 188
at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 145


[Error] Orleans.Runtime.GrainDirectory.ClientDirectory: Exception publishing client routing table to silo

Orleans.Runtime.OrleansMessageRejectionException: The target silo is no longer active: target was ......(silo details).. but this silo is ......(silo details)... The rejected message is Request [......(silo details).. Orleans.Runtime.GrainDirectory.IRemoteClientDirectory.OnUpdateClientRoutes(System.Collections.Immutable.ImmutableDictionary`2[Orleans.Runtime.SiloAddress,System.ValueTuple`2[System.Collections.Immutable.ImmutableHashSet`1[Orleans.Runtime.GrainId],System.Int64]]) #2.
at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98
at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
```
My guess is my grain is not able to communicate with the silos.

Tried Scenarios:

1. Changed the ActivationTimeout for grains to 3 minutes (default from 2 minutes) when configuring the Host Builder.

2. Checked the OnActivateAsync Methods for long running applications.

Kindly help with this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.