System.ObjectDisposedException when restarting TaskHubWorker in DurableTask.AzureStorage
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
Our application is deployed in three regions of Azure and we have written some custom code to handle failover in the case that one of the region goes offline. I can provide more details about this code if necessary, but the exception we are seeing seems to be coming inside DurableTask.AzureStorage, specifically the AppLeaseManager (we currently have App Leases enabled).
Our service is using Microsoft.Azure.DurableTask.AzureStorage v1.9.4 and Microsoft.Azure.DurableTask.Core v2.6.0.
We are able to call StartAsync and then StopAsync on TaskHubWorker instances without issue. However when we call StartAsync again on a TaskHubWorker which was previously started and then stopped, we get the following exception:
at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
at DurableTask.AzureStorage.Partitioning.AppLeaseManager.d__23.MoveNext() in /_/src/DurableTask.AzureStorage/Partitioning/AppLeaseManager.cs:line 123
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.Partitioning.AppLeaseManager.d__21.MoveNext() in /_/src/DurableTask.AzureStorage/Partitioning/AppLeaseManager.cs:line 94
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.AzureStorageOrchestrationService.d__58.MoveNext() in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 397
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.Core.TaskHubWorker.d__22.MoveNext() in C:\source\durabletask\src\DurableTask.Core\TaskHubWorker.cs:line 172
Basically the call to StopAsync disposes of the starterTokenSource and so the subsequent call to StartAsync throws a System.ObjectDisposedExcption when trying to Cancel the same instance of the starterTokenSource. This results in the TaskHubWorker not being started so it is not processing work.
We have worked around this issue by creating a new instance of the AzureStorageOrchestrationService and TaskHubWorker instead of calling StartAsync again, but calling this out here in case calling StartAsync/StopAsync multiple times is expected to succeed.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.