ServiceBus-OperationCanceledException during orchestration transaction

We are getting the following issue when using DTF in azure app service.
DurableTask.Core v2.2.1
DurableTask.ServiceBus v2.1.3

The app service is set to always on with autoscale off and instance count set manually to 1. Orchestrations run successfully for some time (anywhere from 24 hours to 2 weeks), and then we get the below failure. All subsequent orchestrations that are scheduled after the initial failure will also fail with the same pattern. Activities in the orchestration are never scheduled to run and the orchestration message goes to DLQ. Restarting the app service resolves the issue until the next occurrence.

The framework performs the following operations:

env_time | m | e
-- | -- | --
2020-07-21T01:18:17.3195043Z | 1 new messages to process: 489ed065510548d68fa4542e4fbce8e4_d2cd8fbd931547599fcd76b6fe3fcc26, max latency: 6901271.1801ms | ServiceBusOrchestrationService-LockNextTaskOrchestrationWorkItem-MessageToProcess
2020-07-21T01:18:17.5189190Z | No session state exists, creating new session state. | RuntimeStateStreamConverter-GetOrCreateInstanceStateNewSession
2020-07-21T01:18:17.5281970Z | Size of session state is 0, compressed 0 | RuntimeStateStreamConverter-StreamToRuntimeStateSize
2020-07-21T01:18:17.5776820Z | TaskOrchestrationDispatcher-b1e3250f8c4b46f9bbb3d1d12c336353-0: Starting to process workItem 489ed065510548d68fa4542e4fbce8e4 | WorkItemDispatcherProcess-Begin
2020-07-21T01:18:17.5868000Z | Processing new event with Id -1 and type ExecutionStarted | TaskOrchestrationDispatcher-ProcessEvent
2020-07-21T01:18:17.6020908Z | Executed user orchestration. Received 1 orchestrator actions: 0:ScheduleOrchestrator | TaskOrchestrationDispatcher-ExecuteUserOrchestration-End
2020-07-21T01:18:17.6119612Z | Processing orchestrator action of type ScheduleOrchestrator | TaskOrchestrationDispatcher-ProcessOrchestratorAction
2020-07-21T01:18:17.6181143Z | Created new Orchestration Transaction - txnid: b4332235-4efd-4ec0-a4c9-13e03222650c:4077 | ServiceBusOrchestrationService-CompleteTaskOrchestrationWorkItem-CreateTransaction
2020-07-21T01:18:17.6303217Z | Orchestration Transaction Completed b4332235-4efd-4ec0-a4c9-13e03222650c:4077 status: Aborted | ServiceBusOrchestrationService-CompleteTaskOrchestrationWorkItem-TransactionComplete
2020-07-21T01:18:17.6387594Z | TaskOrchestrationDispatcher-b1e3250f8c4b46f9bbb3d1d12c336353-0: Exception while processing workItem 489ed065510548d68fa4542e4fbce8e4\nException: **Microsoft.Azure.ServiceBus.ServiceBusException : The operation was canceled**.\n\t   at Microsoft.Azure.ServiceBus.MessageSession.OnSetStateAsync(Byte[] sessionState)\r\n   at DurableTask.ServiceBus.Common.Abstraction.IMessageSession.SetStateAsync(Byte[] sessionState) in C:\\source\\durabletask\\src\\DurableTask.ServiceBus\\Common\\Abstraction\\ServiceBusAbstraction.cs:line 141\r\n   at DurableTask.ServiceBus.ServiceBusOrchestrationService.TrySetSessionStateAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, OrchestrationRuntimeState runtimeState, IMessageSession session) in C:\\source\\durabletask\\src\\DurableTask.ServiceBus\\ServiceBusOrchestrationService.cs:line 1590\r\n   at DurableTask.ServiceBus.ServiceBusOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in C:\\source\\durabletask\\src\\DurableTask.ServiceBus\\ServiceBusOrchestrationService.cs:line 791\r\n   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in C:\\source\\durabletask\\src\\DurableTask.Core\\TaskOrchestrationDispatcher.cs:line 437\r\n   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in C:\\source\\durabletask\\src\\DurableTask.Core\\TaskOrchestrationDispatcher.cs:line 118\r\n   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in C:\\source\\durabletask\\src\\DurableTask.Core\\WorkItemDispatcher.cs:line 377\nInner Exception: System.OperationCanceledException: The operation was canceled.\r\n   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)\r\n   at Microsoft.Azure.Amqp.SendingAmqpLink.EndSendMessage(IAsyncResult result)\r\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Amqp.Transaction.Controller.DeclareAsync()\r\n   at Microsoft.Azure.ServiceBus.Amqp.AmqpTransactionEnlistment.OnCreateAsync(TimeSpan timeout)\r\n   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)\r\n   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)\r\n   at Microsoft.Azure.ServiceBus.Amqp.AmqpTransactionManager.EnlistAsync(Transaction transaction, ServiceBusConnection serviceBusConnection)\r\n   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ExecuteRequestResponseAsync(AmqpRequestMessage amqpRequestMessage)\r\n   at Microsoft.Azure.ServiceBus.MessageSession.OnSetStateAsync(Byte[] sessionState) | WorkItemDispatcherProcess-Exception
2020-07-21T01:18:17.6459867Z | Backing off after exception by at least 10 until 5 successful operations | WorkItemDispatcherProcess-BackingOff
2020-07-21T01:18:17.6513219Z | Abandoning 1 messages due to work item abort | ServiceBusOrchestrationService-AbandonTaskOrchestrationWorkItem
2020-07-21T01:18:17.7588743Z | DeleteSessionInstance failed, could already be aborted | ServiceBusOrchestrationService-ReleaseTaskOrchestrationWorkItemFailed

Any ideas what could be causing the operation in the thread to stop executing?

Inizia tracciando ServiceBusOrchestrationService.cs alle righe 791 e 1590, ServiceBusAbstraction.cs alla riga 141 e i frame del dispatcher nello stack trace riportato. Determina perché l’impostazione dello stato della sessione annulla la transazione di orchestrazione e perché i messaggi successivi continuano a non riuscire; il lavoro è completato quando la causa è riproducibile e la correzione risultante è verificata.

Scritto dal modello di indicizzazione a partire dal testo della issue.