Orleans Message Rejection Exception on Azure: Unable to Connect to Silo Endpoint

I’m encountering issues with an Orleans application running on Azure. For some odd reason when the silo instances table contains data, I get the following:

```
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.0.0.254:20038:83628028, will retry after 981.0283ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99
at Orleans.Runtime.Messaging.MessageCenter.g__SendAsync|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226
at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
```

Private ports are configured to provide 2 ports and Orleans is configured like this:

```csharp
var endpointAddress =
IPAddress.Parse(context.Configuration["WEBSITE_PRIVATE_IP"]!);
var strPorts =
context.Configuration["WEBSITE_PRIVATE_PORTS"]!.Split(',');
if (strPorts.Length < 2)
throw new Exception("Insufficient private ports configured.");
var (siloPort, gatewayPort) =
(int.Parse(strPorts[0]), int.Parse(strPorts[1]));

siloBuilder
.ConfigureEndpoints(endpointAddress, siloPort, gatewayPort)
.UseAzureStorageClustering(opt =>
{
opt.TableName = orleansSettings.TableStorageConfig.TableName;
opt.TableServiceClient = new TableServiceClient(settings.TableStorageConfig.ConnectionString);
})
.Configure(options =>
{
options.ClusterId = settings.ClusterConfiguration.ClusterId;
options.ServiceId = settings.ClusterConfiguration.ServiceId;
}).AddAzureBlobGrainStorage(Constants.Orleans.GrainStorageProviderName, (AzureBlobStorageOptions options) =>
{
options.BlobServiceClient = new BlobServiceClient(settings.BlobstorageConfig.ConnectionString);
options.ContainerName = Invariant($"{settings.EnvironmentPrefix}{Constants.Orleans.GrainStorageProviderName}");
})
```

Any insights on how I can work past this issue? Besides clearing the instances table?

Start with the reported stack trace in src/Orleans.Core/Networking/ConnectionManager.cs at line 99 and the send path in src/Orleans.Core/Networking/MessageCenter.cs at line 226. Compare the endpoint produced by ConfigureEndpoints with the entries in the Azure silo instances table and the configured WEBSITE_PRIVATE_IP and WEBSITE_PRIVATE_PORTS values; done means identifying why the stored endpoint cannot be reached without clearing the table.

Written by the indexing model from the issue text.