Ungraceful shutdown leading to indefinite startup failures
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 13h 56m
- Merged PRs (30d)
- 351
Description
Observed environment:
- Orleans 3.0.2
- DynamoDB Clustering
- Single silo
- Containerized
Scenario:
- Silo (A) fails in an ungraceful manner (membership entry still marked active)
- New silo starts up
- New silo fails due to https://github.com/dotnet/orleans/issues/4664
- New silos continuing starting and failing until the 10 minute timeout expires
- New silo (B) starts and successfully marks itself as active in the membership table
- Silo B continues startup and executes configured startup tasks (default stage)
- One of the startup tasks makes a grain call
- Grain call tries to active the grain on the dead silo (A)
- Grain activation fails and throws an exception
- The startup task doesn't catch that exception and it bubbles up as an unhandled exception
- Silo (B) process is terminated due to the exception
- Since the shutdown is ungraceful, the loop repeats until membership entries are manually cleaned up
Specific issues:
- The initial failures are by design to account for temporary network outages. I can tweak some configuration options to adjust that window.
- Grain trying to activate on a dead silo. I'm wondering if this is related to https://github.com/dotnet/orleans/issues/5536. Ideally the membership and grain directory could better coordinate in this scenario.
- Startup task failure leading to ungraceful shutdown. I can wrap the startup tasks, but perhaps this scenario could be handled by a more graceful shutdown.
Specific exception trace:
```
Unhandled exception. Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException
: Unable to connect to endpoint S127.0.0.1:22254:417104037. See InnerException
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 127.0.0.1:22254. Error: ConnectionRefused
at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
--- End of inner exception stack trace ---
at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
at Orleans.Runtime.Messaging.OutboundMessageQueue.g__SendAsync|9_0(ValueTask`1 c, Message m)
at Orleans.Runtime.OutgoingCallInvoker.Invoke()
at Orleans.Runtime.OutgoingCallInvoker.Invoke()
at Orleans.Runtime.GrainReferenceRuntime.InvokeWithFilters(GrainReference reference, InvokeMethodRequest request, String debugContext, InvokeMethodOptions options)
at Orleans.Internal.OrleansTaskExtentions.g__ConvertAsync|4_0[T](Task`1 asyncTask)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass36_0.<g__TransportMessageAferSending|0>d.MoveNext()
--- End of stack trace from previous location ---
at Orleans.Runtime.OutgoingCallInvoker.Invoke()
at Orleans.Runtime.OutgoingCallInvoker.Invoke()
at Orleans.Runtime.GrainReferenceRuntime.InvokeWithFilters(GrainReference reference, InvokeMethodRequest request, String debugContext, InvokeMethodOptions options)
at Orleans.Internal.OrleansTaskExtentions.g__ConvertAsync|4_0[T](Task`1 asyncTask)
at PerBlue.Common.GameServer.Grains.Configuration.RuntimeConfigurationService.RequestInitialConfiguration()
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
at Orleans.LifecycleSubject.g__CallOnStart|7_0(Int32 stage, OrderedObserver observer, CancellationToken cancellationToken)
at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
at Orleans.Hosting.SiloWrapper.StartAsync(CancellationToken cancellationToken)
at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at ...
```
Contributor guide
Assessment
This issue has not been assessed yet.