dotnet / dotnet/orleans

In-memory stream stops delivering messages after "Unable to create local activation" and OrleansMessageRejectionException

Open
#9,729 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 57m
Merged PRs (30d)
349

Description

We are using Orleans in-memory streams, and after some time, the streams stop delivering messages. The logs show repeated activation failures for internal memory stream grains, followed by OrleansMessageRejectionException.

Once this occurs, no further messages are delivered to stream subscribers until the silo is restarted.

Logs
```log
[Orleans.Messaging] Failed to forward message Request [S127.0.0.1:11111:119695233 grainid]->[S127.0.0.1:11111:119695233 memorystreamqueue/1067ea7112ad3bf6070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Enqueue(Orleans.Providers.MemoryMessageData) #02F3621706E420C5[ForwardCount=2] from [GrainAddress GrainId memorystreamqueue/1067ea7112ad3bf6070000e007000000, ActivationId: @00000000000000000000000000000000, SiloAddress: S127.0.0.1:11111:119695233] to null after Unable to create local activation. Attempt 2

[Microsoft.Extensions.Logging.ILogger] Exception thrown in MemoryAdapterFactory.QueueMessageBatchAsync.
Orleans.Runtime.OrleansMessageRejectionException: Forwarding failed: tried to forward message Request [S127.0.0.1:11111:119695233 grainid]->[S127.0.0.1:11111:119695233 memorystreamqueue/1067ea7112ad3bf6070000e007000000] Orleans.Providers.IMemoryStreamQueueGrain.Enqueue(Orleans.Providers.MemoryMessageData) #02F3621706E420C5[ForwardCount=2] for 2 times after "Unable to create local activation" to invalid activation. Rejecting now.
at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98
at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Providers.MemoryAdapterFactory`1.QueueMessageBatchAsync[T](StreamId streamId, IEnumerable`1 events, StreamSequenceToken token, Dictionary`2 requestContext) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterFactory.cs:line 162

[Orleans.Storage.MemoryGrainStorage] Error from storage provider MemoryGrainStorage.PubSubRendezvousGrain during WriteStateAsync for grain pubsubrendezvous/StreamProvider/streamid null
Orleans.Runtime.OrleansMessageRejectionException: Forwarding failed: tried to forward message Request [S127.0.0.1:11111:119695233 pubsubrendezvous/StreamProvider/streamId]->[S127.0.0.1:11111:119695233 memorystorage/7] Orleans.Storage.IMemoryStorageGrain.WriteStateAsync(System.String, Orleans.IGrainState`1[System.ReadOnlyMemory`1[System.Byte]]) #02F3621706E420CF[ForwardCount=2] for 2 times after "Unable to create local activation" to invalid activation. Rejecting now.
at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Storage.MemoryGrainStorage.WriteStateAsync[T](String grainType, GrainId grainId, IGrainState`1 grainState) in /_/src/Orleans.Persistence.Memory/Storage/MemoryStorage.cs:line 102
at Orleans.Core.StateStorageBridge`1.WriteStateAsync() in /_/src/Orleans.Runtime/Storage/StateStorageBridge.cs:line 106
```

Contributor guide

Open the contributing guide

Research direction

Start with Orleans.Streaming/MemoryStreams/MemoryAdapterFactory.cs at QueueMessageBatchAsync and Orleans.Persistence.Memory/Storage/MemoryStorage.cs at WriteStateAsync, then reproduce the in-memory stream failure from the reported logs. Trace the activation rejection and verify that stream messages and rendezvous state writes continue after the failure without requiring a silo restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.