dotnet / dotnet/orleans

PubSub Grain hanging

Open
#7,327 12 comments 0 reactions 1 assignee Claimed by @benjaminpetit View on GitHub
area-silos bug
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

Hello folks!

I've captured a weird situation that is throwing warnings and making the the silo to hang:

```
info: Orleans.Runtime.InsideRuntimeClient[0]
Received status update for pending request, Request: Request S192.168.196.95:9591:371069501*grn/SignalR.Orleans.Users.UserGrain/0+RouterClient:ec29d6ec-243c-4fab-9fe6-3c83241c8273@032033ba->S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 InvokeMethodRequest Orleans.Streams.IPubSubRendezvousGrain:UnregisterConsumer #2186. Status: [Activation: S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg-0x36CAD3AA@d8d19e82 #GrainType=Orleans.Streams.PubSubRendezvousGrain Placement=RandomPlacement State=Valid NonReentrancyQueueSize=1 EnqueuedOnDispatcher=0 InFlightCount=1 NumRunning=1 IdlenessTimeSpan=00:02:27.8276600 CollectionAgeLimit=02:00:00]
TaskScheduler status: WorkItemGroup:Name=[Activation: S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 #GrainType=Orleans.Streams.PubSubRendezvousGrain Placement=RandomPlacement State=Valid],WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total Enqueued=27; Total processed=27; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-269:Queued=0; Detailed context=<[Activation: S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg-0x36CAD3AA@d8d19e82 #GrainType=Orleans.Streams.PubSubRendezvousGrain Placement=RandomPlacement State=Valid NonReentrancyQueueSize=1 EnqueuedOnDispatcher=0 InFlightCount=1 NumRunning=1 IdlenessTimeSpan=00:02:27.8276890 CollectionAgeLimit=02:00:00 CurrentlyExecuting=Request S192.168.196.95:9591:371069501*grn/SignalR.Orleans.Users.UserGrain/0+RouterClient:ec29d6ec-243c-4fab-9fe6-3c83241c8273@032033ba->S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 InvokeMethodRequest Orleans.Streams.IPubSubRendezvousGrain:UnregisterConsumer #2186]>
Message Request S192.168.196.95:9591:371069501*grn/SignalR.Orleans.Users.UserGrain/0+RouterClient:ec29d6ec-243c-4fab-9fe6-3c83241c8273@032033ba->S192.168.196.95:9591:371069501*grn/Orleans.Streams.PubSubRendezvousGrain/bdcff7e7-3734-48ab-8599-17d915011b85+ORLEANS_SIGNALR_STREAM_PROVIDER_6ojsbZvH15aP6x79B0JnVg@d8d19e82 InvokeMethodRequest Orleans.Streams.IPubSubRendezvousGrain:UnregisterConsumer #2186 was has been executing for 00:02:27.8245750.
info: Microsoft.Hosting.Lifetime[0]
```

That is happening whenever I unsubscribe to a stream and them subscribe again a few seconds later and unsubscribe.

To give a bit more of context, I have an application that whenever someone connects to SignalR, it calls a grain `OnConnect()` method which in turn subscribe to a set of streams. SignalR.Orleans on its own, also subscribe to a stream. When the client disconnects there is a `OnDisconnect()` call to a grain, which unsubscribe the streams previously subscribed and SignalR.Orleans also unsubscribe to its own stream. There are other many dynamic explicit stream subscriptions that happens as the client move around the world (i.e. it subscribe to the area they are and unsubscribe from others), which is very loaded but for the sake of the tests, I was just trying to connect and disconnect multiple times until the problem happens.

It seems that for whatever reason the`Orleans.Streams.PubSubRendezvousGrain` grain is hanging out and not processing the requests in time.

I have tested with SMS and EventHub stream providers, and also with In-Memory and Azure Tables storage providers for pubsub with Orleans 3.5.0 and .Net 5 on Linux and MacOS.

Do you guys have any idea what may cause that?

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.