Azure / Azure/durabletask

[DurableTask-AzureStorage] Eternal Orchestration Stuck and Consistently Abandoning the Message

Open
#1,019 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

Eternal Orchestration "SiteNetworkServiceStateBillingOrchestrator" started execution and then got stuck while processing a message after lease re-assignment.

The partition "ansmsitenetworkservicehub-control-06" was reassigned to worker node "_armBEaz_11" from worker node "_armBEaz_10", and just after the lease re-assignment, the worker node "armBEaz11" was never able to process one message of (TimerFired Event) and consistently abandoning the message for days.

The orchestration is stuck at line 114 of the code below, note that four task activities were already executed at this point:
![image](https://github.com/Azure/durabletask/assets/47610913/1599f45c-e87a-4b20-a4dd-534772186329)

Logs:
https://jarvis-int-west.microsoftgeneva.com/E06F8A5F
https://jarvis-int-west.microsoftgeneva.com/8D6D9236

Instance Id: 613e83a4-eb15-42c6-aa12-329f0e215894:SiteNetworkServiceStateBillingOrchestrator:V1
Event Type: TimerFired

![image](https://github.com/Azure/durabletask/assets/47610913/b1f13005-d148-423e-b580-aa18d5296f89)
![image](https://github.com/Azure/durabletask/assets/47610913/5d7be30a-7137-495b-8912-2b15ffc84453)

Can someone help identify if this is a race condition? And how we can solve this? This is a billing orchestration which runs periodically, and it is very important to ensure it runs smoothly and consistently emitting billing events.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.