ServiceBus-OperationCanceledException during orchestration transaction

We are getting the following issue when using DTF in azure app service.
DurableTask.Core v2.2.1
DurableTask.ServiceBus v2.1.3

The app service is set to always on with autoscale off and instance count set manually to 1. Orchestrations run successfully for some time (anywhere from 24 hours to 2 weeks), and then we get the below failure. All subsequent orchestrations that are scheduled after the initial failure will also fail with the same pattern. Activities in the orchestration are never scheduled to run and the orchestration message goes to DLQ. Restarting the app service resolves the issue until the next occurrence.

The framework performs the following operations:

env_time | m | e
-- | -- | --
2020-07-21T01:18:17.3195043Z | 1 new messages to process: 489ed065510548d68fa4542e4fbce8e4_d2cd8fbd931547599fcd76b6fe3fcc26, max latency: 6901271.1801ms | ServiceBusOrchestrationService-LockNextTaskOrchestrationWorkItem-MessageToProcess
2020-07-21T01:18:17.5189190Z | No session state exists, creating new session state. | RuntimeStateStreamConverter-GetOrCreateInstanceStateNewSession
2020-07-21T01:18:17.5281970Z | Size of session state is 0, compressed 0 | RuntimeStateStreamConverter-StreamToRuntimeStateSize
2020-07-21T01:18:17.5776820Z | TaskOrchestrationDispatcher-b1e3250f8c4b46f9bbb3d1d12c336353-0: Starting to process workItem 489ed065510548d68fa4542e4fbce8e4 | WorkItemDispatcherProcess-Begin
2020-07-21T01:18:17.5868000Z | Processing new event with Id -1 and type ExecutionStarted | TaskOrchestrationDispatcher-ProcessEvent
2020-07-21T01:18:17.6020908Z | Executed user orchestration. Received 1 orchestrator actions: 0:ScheduleOrchestrator | TaskOrchestrationDispatcher-ExecuteUserOrchestration-End
2020-07-21T01:18:17.6119612Z | Processing orchestrator action of type ScheduleOrchestrator | TaskOrchestrationDispatcher-ProcessOrchestratorAction
2020-07-21T01:18:17.6181143Z | Created new Orchestration Transaction - txnid: b4332235-4efd-4ec0-a4c9-13e03222650c:4077 | ServiceBusOrchestrationService-CompleteTaskOrchestrationWorkItem-CreateTransaction
2020-07-21T01:18:17.6303217Z | Orchestration Transaction Completed b4332235-4efd-4ec0-a4c9-13e03222650c:4077 status: Aborted | ServiceBusOrchestrationService-CompleteTaskOrchestrationWorkItem-TransactionComplete
2020-07-21T01:18:17.6387594Z | TaskOrchestrationDispatcher-b1e3250f8c4b46f9bbb3d1d12c336353-0: Exception while processing workItem 489ed065510548d68fa4542e4fbce8e4\nException: **Microsoft.Azure.ServiceBus.ServiceBusException : The operation was canceled**.\n\t   at Microsoft.Azure.ServiceBus.MessageSession.OnSetStateAsync(Byte[] sessionState)\r\n   at DurableTask.ServiceBus.Common.Abstraction.IMessageSession.SetStateAsync(Byte[] sessionState) in C:\\source\\durabletask\\src\\DurableTask.ServiceBus\\Common\\Abstraction\\ServiceBusAbstraction.cs:line 141\r\n   at DurableTask.ServiceBus.ServiceBusOrchestrationService.TrySetSessionStateAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, OrchestrationRuntimeState runtimeState, IMessageSession session) in C:\\source\\durabletask\\src\\DurableTask.ServiceBus\\ServiceBusOrchestrationService.cs:line 1590\r\n   at DurableTask.ServiceBus.ServiceBusOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in C:\\source\\durabletask\\src\\DurableTask.ServiceBus\\ServiceBusOrchestrationService.cs:line 791\r\n   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in C:\\source\\durabletask\\src\\DurableTask.Core\\TaskOrchestrationDispatcher.cs:line 437\r\n   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in C:\\source\\durabletask\\src\\DurableTask.Core\\TaskOrchestrationDispatcher.cs:line 118\r\n   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in C:\\source\\durabletask\\src\\DurableTask.Core\\WorkItemDispatcher.cs:line 377\nInner Exception: System.OperationCanceledException: The operation was canceled.\r\n   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)\r\n   at Microsoft.Azure.Amqp.SendingAmqpLink.EndSendMessage(IAsyncResult result)\r\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Azure.Amqp.Transaction.Controller.DeclareAsync()\r\n   at Microsoft.Azure.ServiceBus.Amqp.AmqpTransactionEnlistment.OnCreateAsync(TimeSpan timeout)\r\n   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)\r\n   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)\r\n   at Microsoft.Azure.ServiceBus.Amqp.AmqpTransactionManager.EnlistAsync(Transaction transaction, ServiceBusConnection serviceBusConnection)\r\n   at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ExecuteRequestResponseAsync(AmqpRequestMessage amqpRequestMessage)\r\n   at Microsoft.Azure.ServiceBus.MessageSession.OnSetStateAsync(Byte[] sessionState) | WorkItemDispatcherProcess-Exception
2020-07-21T01:18:17.6459867Z | Backing off after exception by at least 10 until 5 successful operations | WorkItemDispatcherProcess-BackingOff
2020-07-21T01:18:17.6513219Z | Abandoning 1 messages due to work item abort | ServiceBusOrchestrationService-AbandonTaskOrchestrationWorkItem
2020-07-21T01:18:17.7588743Z | DeleteSessionInstance failed, could already be aborted | ServiceBusOrchestrationService-ReleaseTaskOrchestrationWorkItemFailed

Any ideas what could be causing the operation in the thread to stop executing?

Beginne damit, ServiceBusOrchestrationService.cs in den Zeilen 791 und 1590, ServiceBusAbstraction.cs in Zeile 141 sowie die Dispatcher-Frames im gemeldeten Stacktrace nachzuverfolgen. Ermittle, warum das Setzen des Sitzungsstatus die Orchestrierungstransaktion abbricht und warum nachfolgende Nachrichten weiterhin fehlschlagen; als erledigt gilt die Aufgabe, wenn die Ursache reproduzierbar ist und die daraus resultierende Korrektur verifiziert wurde.

Vom Indexierungsmodell aus dem Issue-Text verfasst.