PubSub Grain hanging
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
Hello folks!
I've captured a weird situation that is throwing warnings and making the the silo to hang:
```
info: Orleans.Runtime.InsideRuntimeClient[0]
Received status update for pending request, Request: Request S192.168.196.95:9591:371069501*grn/SignalR.Orleans.Users.UserGrain/0+RouterClient:ec29d6ec-243c-4fab-9fe6-3c83241c8273@032033ba->S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 InvokeMethodRequest Orleans.Streams.IPubSubRendezvousGrain:UnregisterConsumer #2186. Status: [Activation: S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg-0x36CAD3AA@d8d19e82 #GrainType=Orleans.Streams.PubSubRendezvousGrain Placement=RandomPlacement State=Valid NonReentrancyQueueSize=1 EnqueuedOnDispatcher=0 InFlightCount=1 NumRunning=1 IdlenessTimeSpan=00:02:27.8276600 CollectionAgeLimit=02:00:00]
TaskScheduler status: WorkItemGroup:Name=[Activation: S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 #GrainType=Orleans.Streams.PubSubRendezvousGrain Placement=RandomPlacement State=Valid],WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total Enqueued=27; Total processed=27; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-269:Queued=0; Detailed context=<[Activation: S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg-0x36CAD3AA@d8d19e82 #GrainType=Orleans.Streams.PubSubRendezvousGrain Placement=RandomPlacement State=Valid NonReentrancyQueueSize=1 EnqueuedOnDispatcher=0 InFlightCount=1 NumRunning=1 IdlenessTimeSpan=00:02:27.8276890 CollectionAgeLimit=02:00:00 CurrentlyExecuting=Request S192.168.196.95:9591:371069501*grn/SignalR.Orleans.Users.UserGrain/0+RouterClient:ec29d6ec-243c-4fab-9fe6-3c83241c8273@032033ba->S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 InvokeMethodRequest Orleans.Streams.IPubSubRendezvousGrain:UnregisterConsumer #2186]>
Message Request S192.168.196.95:9591:371069501*grn/SignalR.Orleans.Users.UserGrain/0+RouterClient:ec29d6ec-243c-4fab-9fe6-3c83241c8273@032033ba->S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 InvokeMethodRequest Orleans.Streams.IPubSubRendezvousGrain:UnregisterConsumer #2186 was has been executing for 00:02:27.8245750.
info: Microsoft.Hosting.Lifetime[0]
```
That is happening whenever I unsubscribe to a stream and them subscribe again a few seconds later and unsubscribe.
To give a bit more of context, I have an application that whenever someone connects to SignalR, it calls a grain `OnConnect()` method which in turn subscribe to a set of streams. SignalR.Orleans on its own, also subscribe to a stream. When the client disconnects there is a `OnDisconnect()` call to a grain, which unsubscribe the streams previously subscribed and SignalR.Orleans also unsubscribe to its own stream. There are other many dynamic explicit stream subscriptions that happens as the client move around the world (i.e. it subscribe to the area they are and unsubscribe from others), which is very loaded but for the sake of the tests, I was just trying to connect and disconnect multiple times until the problem happens.
It seems that for whatever reason the`Orleans.Streams.PubSubRendezvousGrain` grain is hanging out and not processing the requests in time.
I have tested with SMS and EventHub stream providers, and also with In-Memory and Azure Tables storage providers for pubsub with Orleans 3.5.0 and .Net 5 on Linux and MacOS.
Do you guys have any idea what may cause that?
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.