Missed events due to mismatch between PubSubRendevouz and producer state
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
Hello!
We've come across an issue with the pub sub mechanism where in rare situations an explicit stream consumer can end up not receiving any events from a producer until that producer deactivates/reactivates, due to a mismatch between the pub sub state and the producer's internal list of consumers.
When registering a new consumer, PubSubRendezvousGrain persists the subscription before notifying producers of the new subscriber. If the If the call to [producer.AddSubscriber](https://github.com/dotnet/orleans/blob/main/src/Orleans.Streaming/PubSub/PubSubRendezvousGrain.cs#L210) fails for any reason other than a select few (GrainExtensionNotInstalledException, ClientNotAvailableException, or OrleansMessageRejectionException if on a system target), then the subscription will remain in the pub sub state even though the subscribe operation will appear to fail. We end up in a situation where the subscription appears to exist, but the producer doesn't know about it. This means the consumer grain could get the handle and resume it successfully, but the producer would still be unaware of and fail to deliver events to the consumer.
There's a minimal repro demonstrating the issue here: [OrleansPubSub-MissedEvents](https://github.com/mindlink/OrleansPubSub-MissedEvents/)
The repro uses a grain call filter to force the producer.AddSubscriber call to throw.
Note: We are using Orleans 3.7.2, but believe this may be an issue with later versions of Orleans as well.
Contributor guide
Research direction
Start with src/Orleans.Streaming/PubSub/PubSubRendezvousGrain.cs around the producer.AddSubscriber call at line 210, then run the OrleansPubSub-MissedEvents minimal repro and inspect its grain call filter. Trace the subscription persistence and notification failure path; done means a failed registration cannot leave pub/sub state claiming a consumer that the producer does not know about, with the repro no longer missing events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100