dotnet / dotnet/orleans

InvalidOperationException with message "Block is full"

Open
#7,536 14 comments 1 reaction 1 assignee Claimed by @benjaminpetit View on GitHub
area-streaming Needs: attention :wave:
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

Hi guys,

First of all, thank you very much for the wonderful work ! Orleans is so useful and dev time saving !
That said, my team and I found 5.2K occurences of a `System.InvalidOperationException` with this error message : "Block is full"

Here is the stack trace:

```
System.InvalidOperationException:
at Orleans.Providers.Streams.Common.PooledQueueCache.Add (OrleansProviders, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
at Orleans.ServiceBus.Providers.EventHubQueueCache.Add (Orleans.Streaming.EventHubs, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
at Orleans.ServiceBus.Providers.EventHubAdapterReceiver+d__22.MoveNext (Orleans.Streaming.EventHubs, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Orleans.Streams.PersistentStreamPullingAgent+d__37.MoveNext (Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Orleans.Internal.AsyncExecutorWithRetries+d__4`1.MoveNext (Orleans.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Orleans.Internal.AsyncExecutorWithRetries+d__4`1.MoveNext (Orleans.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Orleans.Streams.PersistentStreamPullingAgent+d__35.MoveNext (Orleans.Runtime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
```

The formatted message was :
`Giving up reading from queue my-stream-0 after retry attempts 6`

We are using Orleans v3.5.0
We don't have an explanation why this exception has been raised or what could have triggered it.
We analyzed Orleans' code but, beside finding where it is actually raised, we could not figure out a way where this case could occur.

Do you have any idea on your side ?

N.B.: it is a repost from your Discord asked by @ReubenBond

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.