[DurableTask-AzureStorage] Eternal Orchestration Stuck and Consistently Abandoning the Message
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
Eternal Orchestration "SiteNetworkServiceStateBillingOrchestrator" started execution and then got stuck while processing a message after lease re-assignment.
The partition "ansmsitenetworkservicehub-control-06" was reassigned to worker node "_armBEaz_11" from worker node "_armBEaz_10", and just after the lease re-assignment, the worker node "armBEaz11" was never able to process one message of (TimerFired Event) and consistently abandoning the message for days.
The orchestration is stuck at line 114 of the code below, note that four task activities were already executed at this point:

Logs:
https://jarvis-int-west.microsoftgeneva.com/E06F8A5F
https://jarvis-int-west.microsoftgeneva.com/8D6D9236
Instance Id: 613e83a4-eb15-42c6-aa12-329f0e215894:SiteNetworkServiceStateBillingOrchestrator:V1
Event Type: TimerFired


Can someone help identify if this is a race condition? And how we can solve this? This is a billing orchestration which runs periodically, and it is very important to ensure it runs smoothly and consistently emitting billing events.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.