Orleans 3.2.0 EventHub user errors (Exception while retrying the 1th time reading from queue ppprovider-2) after Silos restart (Context: I have been told I am dead, so this silo will stop! I should be Dead according to membership table)

Orleans v3.2.0, on .NET Core 3.1 - EventHub User Errors

Hi,

We are experiencing User Errors lately on EventHub.

We have 3 Silos deployed as Windows Services using VMSS (& Topshelf)
We have a web app that is writing events to EventHub (configured via Orleans) and the Grains within the Silos are consuming the messages.

(EventHub has 1 consumer group and 16 partitions).

We see User Errors in EventHub, and in AppInsights errors like these:
Exception while retrying the 1th time reading from queue ppprovider-2
The supplied offset '9294315870216' is invalid. The last offset in the system is '2130312930776' T

What we've noticed is that every 2-3 days, 1 or 2 Silos (Windows Services) are stopping. **(exactly at 12 AM)**
EventViews logs:

Application: []Run.exe
CoreCLR Version: 4.700.22.55902
.NET Core Version: 3.1.32
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: FATAL EXCEPTION from Orleans.Runtime.MembershipService.MembershipTableManager. Context: I have been told I am dead, so this silo will stop! I should be Dead according to membership table (in TryToSuspectOrKill)

We have a temporarly Scheduled Task that checks if the services are stopped and restarts them.

Usually, these UserErrors are thrown after the restart happens.

If we delete the checkpointers from Storage and restart the Silos, errors are no more thrown and process works normally.

What we managed to find from our Azure logs:
- VMSS are healthy
- Orleans - heartbit checks - Silo 1, Silo 2, Silo 3; Silo 1 tries to call Silo 2, no response => mark it as dead, Silo 3 tries to call Silo 1, no response => mark it as dead. Silo 3 is the master one.

We would like to understand:
**1. Are these users errrors from EventHub correlated with the Silos being restarted?
2. Why the Windows Services stop?**

Start by correlating the Orleans MembershipTableManager/TryToSuspectOrKill shutdown logs with the EventHub checkpoint records and silo restart times. Compare the invalid offsets before and after checkpoint deletion, then document whether the restart causes the errors and what explains the Windows Service termination.

Written by the indexing model from the issue text.