Azure / Azure/durabletask

TaskCancelledException in RemoteOrchestrationServiceClient

Open
#1,156 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

We are using SF backend and getting TaskCancelledException while trying to resolve the partitions in RemoteOrchestrationClient in 1 of the many Prod regions. We have an http timeout of 10sec while creating the TaskhubClient and those time out too. This fails to start any DTF operation in that region and fails the Task.
This is the stack trace:
`{"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing.","Data":null,"InnerException":{"ClassName":"System.TimeoutException","Message":"A task was canceled.","Data":null,"InnerException":{"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233029,"Source":"System.Private.CoreLib","WatsonBuckets":null},"HelpURL":null,"StackTraceString":null,"RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2146233083,"Source":null,"WatsonBuckets":null},"HelpURL":null,"StackTraceString":" at DurableTask.AzureServiceFabric.Remote.RemoteOrchestrationServiceClient.ExecuteRequestWithRetriesAsync(String instanceId, Func2 requestAsync, CancellationToken cancellationToken) in C:\\__w\\1\\s\\src\\Services\\DurableTask.AzureServiceFabric\\Remote\\RemoteOrchestrationServiceClient.cs:line 403\r\n at DurableTask.AzureServiceFabric.Remote.RemoteOrchestrationServiceClient.PutJsonAsync(String instanceId, String fragment, Object object, CancellationToken cancellationToken) in C:\\__w\\1\\s\\src\\Services\\DurableTask.AzureServiceFabric\\Remote\\RemoteOrchestrationServiceClient.cs:line 317\r\n at DurableTask.AzureServiceFabric.Remote.RemoteOrchestrationServiceClient.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in C:\\__w\\1\\s\\src\\Services\\DurableTask.AzureServiceFabric\\Remote\\RemoteOrchestrationServiceClient.cs:line 101\r\n at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 622`

What is a possible solution to this issue? Do we need additional retries while trying to start the Orchestration, so that it retries the request on a different partition?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.