In-memory stream stops delivering messages after "Unable to create local activation" and OrleansMessageRejectionException
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 57m
- Merged PRs (30d)
- 349
Description
We are using Orleans in-memory streams, and after some time, the streams stop delivering messages. The logs show repeated activation failures for internal memory stream grains, followed by OrleansMessageRejectionException.
Once this occurs, no further messages are delivered to stream subscribers until the silo is restarted.
Logs
```log
[Orleans.Messaging] Failed to forward message Request [S127.0.0.1:11111:119695233 grainid]->[S127.0.0.1:11111:119695233 memorystreamqueue/1067ea7112ad3bf6070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Enqueue(Orleans.Providers.MemoryMessageData) #02F3621706E420C5[ForwardCount=2] from [GrainAddress GrainId memorystreamqueue/1067ea7112ad3bf6070000e007000000, ActivationId: @00000000000000000000000000000000, SiloAddress: S127.0.0.1:11111:119695233] to null after Unable to create local activation. Attempt 2
[Microsoft.Extensions.Logging.ILogger] Exception thrown in MemoryAdapterFactory.QueueMessageBatchAsync.
Orleans.Runtime.OrleansMessageRejectionException: Forwarding failed: tried to forward message Request [S127.0.0.1:11111:119695233 grainid]->[S127.0.0.1:11111:119695233 memorystreamqueue/1067ea7112ad3bf6070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Enqueue(Orleans.Providers.MemoryMessageData) #02F3621706E420C5[ForwardCount=2] for 2 times after "Unable to create local activation" to invalid activation. Rejecting now.
at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98
at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Providers.MemoryAdapterFactory`1.QueueMessageBatchAsync[T](StreamId streamId, IEnumerable`1 events, StreamSequenceToken token, Dictionary`2 requestContext) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterFactory.cs:line 162
[Orleans.Storage.MemoryGrainStorage] Error from storage provider MemoryGrainStorage.PubSubRendezvousGrain during WriteStateAsync for grain pubsubrendezvous/StreamProvider/streamid null
Orleans.Runtime.OrleansMessageRejectionException: Forwarding failed: tried to forward message Request [S127.0.0.1:11111:119695233 pubsubrendezvous/StreamProvider/streamId]->[S127.0.0.1:11111:119695233 memorystorage/7] Orleans.Storage.IMemoryStorageGrain.WriteStateAsync(System.String, Orleans.IGrainState`1[System.ReadOnlyMemory`1[System.Byte]]) #02F3621706E420CF[ForwardCount=2] for 2 times after "Unable to create local activation" to invalid activation. Rejecting now.
at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Storage.MemoryGrainStorage.WriteStateAsync[T](String grainType, GrainId grainId, IGrainState`1 grainState) in /_/src/Orleans.Persistence.Memory/Storage/MemoryStorage.cs:line 102
at Orleans.Core.StateStorageBridge`1.WriteStateAsync() in /_/src/Orleans.Runtime/Storage/StateStorageBridge.cs:line 106
```
Contributor guide
Research direction
Start with Orleans.Streaming/MemoryStreams/MemoryAdapterFactory.cs at QueueMessageBatchAsync and Orleans.Persistence.Memory/Storage/MemoryStorage.cs at WriteStateAsync, then reproduce the in-memory stream failure from the reported logs. Trace the activation rejection and verify that stream messages and rendezvous state writes continue after the failure without requiring a silo restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100