Azure / Azure/durabletask

System.ObjectDisposedException when restarting TaskHubWorker in DurableTask.AzureStorage

Open
#675 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

Our application is deployed in three regions of Azure and we have written some custom code to handle failover in the case that one of the region goes offline. I can provide more details about this code if necessary, but the exception we are seeing seems to be coming inside DurableTask.AzureStorage, specifically the AppLeaseManager (we currently have App Leases enabled).

Our service is using Microsoft.Azure.DurableTask.AzureStorage v1.9.4 and Microsoft.Azure.DurableTask.Core v2.6.0.

We are able to call StartAsync and then StopAsync on TaskHubWorker instances without issue. However when we call StartAsync again on a TaskHubWorker which was previously started and then stopped, we get the following exception:

at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
at DurableTask.AzureStorage.Partitioning.AppLeaseManager.d__23.MoveNext() in /_/src/DurableTask.AzureStorage/Partitioning/AppLeaseManager.cs:line 123
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.Partitioning.AppLeaseManager.d__21.MoveNext() in /_/src/DurableTask.AzureStorage/Partitioning/AppLeaseManager.cs:line 94
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.AzureStorageOrchestrationService.d__58.MoveNext() in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 397
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.Core.TaskHubWorker.d__22.MoveNext() in C:\source\durabletask\src\DurableTask.Core\TaskHubWorker.cs:line 172

Basically the call to StopAsync disposes of the starterTokenSource and so the subsequent call to StartAsync throws a System.ObjectDisposedExcption when trying to Cancel the same instance of the starterTokenSource. This results in the TaskHubWorker not being started so it is not processing work.

We have worked around this issue by creating a new instance of the AzureStorageOrchestrationService and TaskHubWorker instead of calling StartAsync again, but calling this out here in case calling StartAsync/StopAsync multiple times is expected to succeed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.