stuck eternal orchestration and unable to terminate unless the service is restarted

This is an issue with DurableTask.AzureStorage v1.8.7.
Some already running eternal orchestration got stuck.
We started seeing a lot of logs such as "Received a duplicate message [___#0] from main-control-03 with ID = ___ and dequeue count = _".
It seems to be inside an infinite loop with dequeue count continuing to increase.
This also pegged the CPU to 100 %.
We attempted many times to terminate the orchestration via .net API, but the orchestration would not terminate.
We had to restart the app service instance on Azure and then after instance restarted, orchestration was terminated and restarted successfully, but it then self-terminated probably because it started replaying all the terminate messages that were attempted before restarting the app instance.
Realizing after a day that orchestration has not been running at all, we were able to terminate and restart it successfully.

Some logs when it got stuck:

a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#5] from main-control-03 with ID = 4db9d87f-a377-4217-9678-b162fd36c376 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#15] from main-control-03 with ID = bcda6cd8-2e7d-4d5b-a57b-e2fa280cd8da and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#13] from main-control-03 with ID = e8762bed-dc53-41be-afe4-93f6263ab2e6 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#8] from main-control-03 with ID = 58526770-5fa7-4ee9-bf4f-bc1cd1c3da7c and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#14] from main-control-03 with ID = 7f3f176d-a334-4342-8933-3b5a5c5132e4 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#6] from main-control-03 with ID = dc9703bf-2531-4fee-b295-78e12b44b931 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#16] from main-control-03 with ID = e0217340-649c-4ad7-9f40-397b822b7151 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#7] from main-control-03 with ID = 03a2c9fa-d0c4-4871-9454-4d265830169f and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#17] from main-control-03 with ID = 247e7eb5-0a3c-4699-b2a0-e4ceef8df96a and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#11] from main-control-03 with ID = 95cad5d9-cfae-46ab-83ee-758e37c5ee97 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#9] from main-control-03 with ID = 152214f7-76af-4aa4-aed9-484481efb123 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#10] from main-control-03 with ID = 0e3b63f7-a2ce-417b-9ce4-edb927213011 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#12] from main-control-03 with ID = e07aec82-e788-4e13-bb1e-3ce2932ad4e9 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#5] message from main-control-03 (delay = 313767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#15] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#13] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#16] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#14] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#6] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#8] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#7] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#17] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#11] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#9] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#10] message from main-control-03 (delay = 312767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#12] message from main-control-03 (delay = 310767ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 1819924ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched 35 history events from the history table in 15ms
85d1f6ce-afd4-4d2e-a72a-e5573767ec36: Discarding 1 dequeued history event(s): Invalid history (may have been overwritten by a newer instance)
main-applease: Failed to acquire app lease with appLeaseId efa66418-1043-42cb-d8f5-52fe05849113. Another app likely has the lease on this container. Exception: There is already a lease present.
main-applease: Failed to acquire app lease with appLeaseId efa66418-1043-42cb-d8f5-52fe05849113. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 3beaedab-b903-4ed6-bf6a-27c3c9748065 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2137875ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: No history events were found
main2-applease: Failed to acquire app lease with appLeaseId 2a6c3454-f98a-3bdf-a6fc-f94f5cf81986. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 1820547ms)
main2-applease: Failed to acquire app lease with appLeaseId 2a6c3454-f98a-3bdf-a6fc-f94f5cf81986. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#7] from main-control-03 with ID = 03a2c9fa-d0c4-4871-9454-4d265830169f and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#10] from main-control-03 with ID = 0e3b63f7-a2ce-417b-9ce4-edb927213011 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#9] from main-control-03 with ID = 152214f7-76af-4aa4-aed9-484481efb123 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#17] from main-control-03 with ID = 247e7eb5-0a3c-4699-b2a0-e4ceef8df96a and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#5] from main-control-03 with ID = 4db9d87f-a377-4217-9678-b162fd36c376 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#8] from main-control-03 with ID = 58526770-5fa7-4ee9-bf4f-bc1cd1c3da7c and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#14] from main-control-03 with ID = 7f3f176d-a334-4342-8933-3b5a5c5132e4 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#11] from main-control-03 with ID = 95cad5d9-cfae-46ab-83ee-758e37c5ee97 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#15] from main-control-03 with ID = bcda6cd8-2e7d-4d5b-a57b-e2fa280cd8da and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#6] from main-control-03 with ID = dc9703bf-2531-4fee-b295-78e12b44b931 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#16] from main-control-03 with ID = e0217340-649c-4ad7-9f40-397b822b7151 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#12] from main-control-03 with ID = e07aec82-e788-4e13-bb1e-3ce2932ad4e9 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#13] from main-control-03 with ID = e8762bed-dc53-41be-afe4-93f6263ab2e6 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#5] message from main-control-03 (delay = 616027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#15] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#13] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#16] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#14] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#6] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#8] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#7] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#17] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#11] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#9] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#10] message from main-control-03 (delay = 615027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#12] message from main-control-03 (delay = 613027ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 41b50a5b-41b8-4425-a855-3ed5667c8cb6 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2124937ms)
85d1f6ce-afd4-4d2e-a72a-e5573767ec36: Discarding 1 dequeued history event(s): Invalid history (may have been overwritten by a newer instance)
main-applease: Failed to acquire app lease with appLeaseId efa66418-1043-42cb-d8f5-52fe05849113. Another app likely has the lease on this container. Exception: There is already a lease present.
main-applease: Failed to acquire app lease with appLeaseId efa66418-1043-42cb-d8f5-52fe05849113. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 3beaedab-b903-4ed6-bf6a-27c3c9748065 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2438268ms)
main2-applease: Failed to acquire app lease with appLeaseId 2a6c3454-f98a-3bdf-a6fc-f94f5cf81986. Another app likely has the lease on this container. Exception: There is already a lease present.
main2-applease: Failed to acquire app lease with appLeaseId 2a6c3454-f98a-3bdf-a6fc-f94f5cf81986. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 7706cf4d-a7fe-447f-a8b1-e41db49e4007 and dequeue count = 2
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2121627ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#7] from main-control-03 with ID = 03a2c9fa-d0c4-4871-9454-4d265830169f and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#10] from main-control-03 with ID = 0e3b63f7-a2ce-417b-9ce4-edb927213011 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#9] from main-control-03 with ID = 152214f7-76af-4aa4-aed9-484481efb123 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#17] from main-control-03 with ID = 247e7eb5-0a3c-4699-b2a0-e4ceef8df96a and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#5] from main-control-03 with ID = 4db9d87f-a377-4217-9678-b162fd36c376 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#8] from main-control-03 with ID = 58526770-5fa7-4ee9-bf4f-bc1cd1c3da7c and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#14] from main-control-03 with ID = 7f3f176d-a334-4342-8933-3b5a5c5132e4 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#11] from main-control-03 with ID = 95cad5d9-cfae-46ab-83ee-758e37c5ee97 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#15] from main-control-03 with ID = bcda6cd8-2e7d-4d5b-a57b-e2fa280cd8da and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#6] from main-control-03 with ID = dc9703bf-2531-4fee-b295-78e12b44b931 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#16] from main-control-03 with ID = e0217340-649c-4ad7-9f40-397b822b7151 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#12] from main-control-03 with ID = e07aec82-e788-4e13-bb1e-3ce2932ad4e9 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#13] from main-control-03 with ID = e8762bed-dc53-41be-afe4-93f6263ab2e6 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#5] message from main-control-03 (delay = 916156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#15] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#13] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#16] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#14] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#6] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#8] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#7] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#17] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#11] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#9] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#10] message from main-control-03 (delay = 915156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TaskCompleted#12] message from main-control-03 (delay = 913156ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 41b50a5b-41b8-4425-a855-3ed5667c8cb6 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2452893ms)
main-applease: Failed to acquire app lease with appLeaseId efa66418-1043-42cb-d8f5-52fe05849113. Another app likely has the lease on this container. Exception: There is already a lease present.
main-applease: Failed to acquire app lease with appLeaseId efa66418-1043-42cb-d8f5-52fe05849113. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 3beaedab-b903-4ed6-bf6a-27c3c9748065 and dequeue count = 4
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2764239ms)
main2-applease: Failed to acquire app lease with appLeaseId 2a6c3454-f98a-3bdf-a6fc-f94f5cf81986. Another app likely has the lease on this container. Exception: There is already a lease present.
main2-applease: Failed to acquire app lease with appLeaseId 2a6c3454-f98a-3bdf-a6fc-f94f5cf81986. Another app likely has the lease on this container. Exception: There is already a lease present.
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TimerFired#0] from main-control-03 with ID = 7706cf4d-a7fe-447f-a8b1-e41db49e4007 and dequeue count = 3
a266bdb3-a656-4f7a-befb-483fb5dc0527: Fetched [TimerFired#0] message from main-control-03 (delay = 2425002ms)
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#7] from main-control-03 with ID = 03a2c9fa-d0c4-4871-9454-4d265830169f and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#10] from main-control-03 with ID = 0e3b63f7-a2ce-417b-9ce4-edb927213011 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#9] from main-control-03 with ID = 152214f7-76af-4aa4-aed9-484481efb123 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#17] from main-control-03 with ID = 247e7eb5-0a3c-4699-b2a0-e4ceef8df96a and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#5] from main-control-03 with ID = 4db9d87f-a377-4217-9678-b162fd36c376 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#8] from main-control-03 with ID = 58526770-5fa7-4ee9-bf4f-bc1cd1c3da7c and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#14] from main-control-03 with ID = 7f3f176d-a334-4342-8933-3b5a5c5132e4 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#11] from main-control-03 with ID = 95cad5d9-cfae-46ab-83ee-758e37c5ee97 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#15] from main-control-03 with ID = bcda6cd8-2e7d-4d5b-a57b-e2fa280cd8da and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#6] from main-control-03 with ID = dc9703bf-2531-4fee-b295-78e12b44b931 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#16] from main-control-03 with ID = e0217340-649c-4ad7-9f40-397b822b7151 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#12] from main-control-03 with ID = e07aec82-e788-4e13-bb1e-3ce2932ad4e9 and dequeue count = 5
a266bdb3-a656-4f7a-befb-483fb5dc0527: Received a duplicate message [TaskCompleted#13] from main-control-03 with ID = e8762bed-dc53-41be-afe4-93f6263ab2e6 and dequeue count = 5

Inizia con DurableTask.AzureStorage v1.8.7 e l’API di terminazione .NET, usando i log duplicate-message, invalid-history, CPU e app-lease come percorso di errore osservato. Traccia il modo in cui l’orchestrazione gestisce i messaggi TaskCompleted e TimerFired ripetuti e le richieste di terminazione; il lavoro è completato quando un’orchestrazione eterna bloccata può terminare senza riavviare l’Azure app service.

Scritto dal modello di indicizzazione a partire dal testo della issue.