RunAsync in service abstractions doesn't handle failing actions correctly
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 77
- Forks
- 40
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 10
Description
The use of WhenAll in https://github.com/microsoft/Omex/blob/main/src/Hosting.Services/OmexStatelessService.cs and it's Stateful equivalent means that if an action throws, the action ends but the service replica is not restarted, since WhenAll waits for all other tasks to end also.
This prevents registering of more than one action which should cause a service replica restart on failure and leaves the developer with only two options in such cases:
- Swallow exceptions, log failures and rely passive monitoring to alert on the service failures. This would require manual intervention to recover.
- Add complex task management logic to the service code, wrapping multiple actions into a single action for registration with the service instance.
Neither of these options is a good solution.
A better approach would be to update these service classes to apply WhenAny to the enumeration of tasks.
If any task throws a non-recoverable exception, an appropriate exception would be thrown here to cause the service replica to recycle and the remaining tasks would be cancelled/ended.
If a single task ends gracefully, remaining tasks are waited using WhenAny again.
This would allow for a single action throwing to trigger service recovery, while also allowing for graceful ending of actions.
Relevant reference docs:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statelessservice.runasync?view=azure-dotnet#remarks
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statelessservice.runasync?view=azure-dotnet#microsoft-servicefabric-services-runtime-statelessservice-runasync(system-threading-cancellationtoken)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/Hosting.Services/OmexStatelessService.cs and its Stateful equivalent, at the RunAsync implementation that currently uses WhenAll. Read the linked Service Fabric RunAsync remarks, then verify that a failing action triggers replica recovery while graceful action completion allows remaining actions to end; account for cancellation and confirm behavior for multiple registered actions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100