dotnet / dotnet/orleans

Missed events due to mismatch between PubSubRendevouz and producer state

Open
#9,739 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

Hello!

We've come across an issue with the pub sub mechanism where in rare situations an explicit stream consumer can end up not receiving any events from a producer until that producer deactivates/reactivates, due to a mismatch between the pub sub state and the producer's internal list of consumers.

When registering a new consumer, PubSubRendezvousGrain persists the subscription before notifying producers of the new subscriber. If the If the call to [producer.AddSubscriber](https://github.com/dotnet/orleans/blob/main/src/Orleans.Streaming/PubSub/PubSubRendezvousGrain.cs#L210) fails for any reason other than a select few (GrainExtensionNotInstalledException, ClientNotAvailableException, or OrleansMessageRejectionException if on a system target), then the subscription will remain in the pub sub state even though the subscribe operation will appear to fail. We end up in a situation where the subscription appears to exist, but the producer doesn't know about it. This means the consumer grain could get the handle and resume it successfully, but the producer would still be unaware of and fail to deliver events to the consumer.

There's a minimal repro demonstrating the issue here: [OrleansPubSub-MissedEvents](https://github.com/mindlink/OrleansPubSub-MissedEvents/)
The repro uses a grain call filter to force the producer.AddSubscriber call to throw.

Note: We are using Orleans 3.7.2, but believe this may be an issue with later versions of Orleans as well.

Contributor guide

Open the contributing guide

Research direction

Start with src/Orleans.Streaming/PubSub/PubSubRendezvousGrain.cs around the producer.AddSubscriber call at line 210, then run the OrleansPubSub-MissedEvents minimal repro and inspect its grain call filter. Trace the subscription persistence and notification failure path; done means a failed registration cannot leave pub/sub state claiming a consumer that the producer does not know about, with the repro no longer missing events.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.