dotnet / dotnet/orleans

Lost stream messages

Open
#9,828 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

Observation:
Rarely stream messages can fail to be delivered to an Orleans client. Persists until silos are redeployed.

Details:
* End user connects to Orleans Client Application via API
* Upon connection, the Orleans client subscribes to a stream for that user
* The stream subscription is used by Orleans Silos to send messages back to the end user
* It's possible for a particular stream to stop delivering messages despite reconnection / resubscription. Only fixed on silo redeployment.

Configuration:
* Orleans 9.2.1
* In memory streams (`AddMemoryStreams("MemProvider"`)
* DynamoDB pubsub storage
* Activation repartitioner enabled (defaults)
* Activation rebalancer enabled (defaults)

Example troubleshooting sequence (from telemetry)
1. Some amount of end user connections causing stream subscriptions / silo changes
* Basically the cluster had been running for awhile and the end user had come and gone. I can try to pull more details if it seems relevant
2. End user connects and subscribes to the stream
3. At this point there were 2 producers registered.
* One of which was removed as a result of a failed `Orleans.Streams.IStreamProducerExtension.AddSubscriber` call (I believe that particular silo instance was no longer active based on the ip in the sys target).
* The other producer successfully added the subscriber
4. A short while later an event occurred which generated a new message on the stream
* Can see that `IMemoryStreamQueueGrain.Enqueue` was called
5. Shortly after can see a call to `IMemoryStreamQueueGrain.Dequeue` for the same grain
* No calls to `IStreamConsumerExtension.DeliverBatch` or `IPubSubRendezvousGrain.RegisterProducer` observed
6. Future stream messages also not delivered despite more connection / subscription attempts
7. After a silo deployment, stream messages for that particular user stream began to function normally again

I'm wondering if the `pubSubCache` within the `PersistentStreamPullingAgent` was stale in some way that it thought all information was up to date but actually it wasn't.

Contributor guide

Open the contributing guide

Research direction

Start by tracing PersistentStreamPullingAgent and its pubSubCache around the reported IMemoryStreamQueueGrain.Enqueue/Dequeue sequence. Inspect the IStreamProducerExtension.AddSubscriber, IStreamConsumerExtension.DeliverBatch, and IPubSubRendezvousGrain.RegisterProducer calls; done means identifying why delivery stops until silo redeployment and documenting a reproducible fix or regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.