dotnet / dotnet/orleans

Azure App Service fails to start with never ending Orleans connection exceptions

Open
#8,891 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

We have an Azure App Service running Orleans 7.2.4 with ASP.NET Core on .NET 8. It has been running well for a few weeks. This morning after a scheduled upscale the site failed to start and logs exceptions like this:

```
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.0.2.254:20016:68181414, will retry after 454.0547ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99
at Orleans.Runtime.Messaging.MessageCenter.g__SendAsync|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226

Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.0.2.254:20016. Error: AccessDenied
Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.0.2.254:20024. Error: AccessDenied

Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.0.2.254:20024:68182019. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.0.2.254:20024. Error: AccessDenied
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 54
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 221
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 106
at Orleans.Runtime.Messaging.MessageCenter.g__SendAsync|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226
```

These happened continuously for hours. We had to remove Orleans before the site would start up.

Private ports are configured to provide 2 ports. Orleans is configured like this:
```C#
var endpointAddress = IPAddress.Parse(context.Configuration["WEBSITE_PRIVATE_IP"]);
var strPorts = context.Configuration["WEBSITE_PRIVATE_PORTS"].Split(',');
var siloPort = int.Parse(strPorts[0]);
var gatewayPort = int.Parse(strPorts[1]);

siloBuilder
.UseAzureStorageClustering(options =>
{
options.TableName = orleansSettings.TableStorageConfig.TableName;
options.ConfigureTableServiceClient(orleansSettings.TableStorageConfig.ConnectionString);
})
.ConfigureEndpoints(endpointAddress, siloPort, gatewayPort)
.Configure(options =>
{
options.ClusterId = orleansSettings.ClusterConfig.ClusterId;
options.ServiceId = orleansSettings.ClusterConfig.ServiceId;
})
.AddAzureBlobGrainStorage(DefaultGrainStorageProvider, options =>
{
options.ConfigureBlobServiceClient(orleansSettings.BlobStorageConfig.ConnectionString);
options.ContainerName = orleansSettings.BlobStorageConfig.ContainerName;
});
```

We also use Orleans Dashboard. We do not use reminders or streams.

The App Service has only 1 instance and is the only member of the cluster.
The clustering table has about 100 rows in it. The Status of some rows is Dead but many more are Joining or Active.

Changing to a different clustering table didn't resolve the issue.

Contributor guide

Open the contributing guide

Research direction

Start with the reported stack traces in ConnectionManager.cs, MessageCenter.cs, SocketConnectionFactory.cs, and ConnectionFactory.cs, then compare them with the WEBSITE_PRIVATE_IP and WEBSITE_PRIVATE_PORTS endpoint configuration. Inspect the clustering table rows and Azure App Service networking context described in the report. Done means identifying the cause of the AccessDenied connection failures and a verified way for the single-instance site to start with Orleans enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.