DurableTaskStorageException: The client could not finish the operation within specified timeout

I am looking for any leads with respect to the following error that I am observing that does not appear to affect my code (i.e., code appears to continue to function):

`Failed during ownership acquisition: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The client could not finish the operation within specified timeout`

This error does not appear on a regular basis (meaning that it can be hours before occurring again and other times within 30 minutes).

The full stack trace is as follows:

```
Failed during ownership acquisition: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The client could not finish the operation within specified timeout.
---> Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout.
---> System.TimeoutException: The client could not finish the operation within specified timeout.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
---> System.Net.Sockets.SocketException (125): Operation canceled
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable`1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)
at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
Request Information
RequestID:
RequestDate:
StatusMessage:
ErrorCode:

--- End of inner exception stack trace ---
at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 140
at DurableTask.AzureStorage.Storage.BlobContainer.ListBlobsAsync(String blobDirectory) in /_/src/DurableTask.AzureStorage/Storage/BlobContainer.cs:line 106
at DurableTask.AzureStorage.Partitioning.BlobLeaseManager.ListLeasesAsync() in /_/src/DurableTask.AzureStorage/Partitioning/BlobLeaseManager.cs:line 79
at DurableTask.AzureStorage.Partitioning.LeaseCollectionBalancer`1.TakeLeasesAsync()
at DurableTask.AzureStorage.Partitioning.LeaseCollectionBalancer`1.LeaseTakerAsync()
```

The source of the error is located [in the AzureStorageClient](https://github.com/Azure/durabletask/blob/619cdbfc62ce6b57d3d1b34e15068eb0e62bec0f/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs#L140), but it is unclear why this is happening.

Note that the `RequestID`, `RequestDate`, `StatusMessage` and `ErrorCode` are empty.

The `EventType` that is logged is "PartitionManagerError"

Given the stack trace, and the fact that the event type is "PartitionManagerError", I believe that the source of the error is in the [LeaseCollectionBalancer](https://github.com/Azure/durabletask/blob/d215370465d5494df2a8da679b15f08bd212d834/src/DurableTask.AzureStorage/Partitioning/LeaseCollectionBalancer.cs#L284). If you look, when an exception occurs, then a `PartitionManagerError` is logged.

In between instances of the above error, the following error is also logged:

`Failed during intent acquisition: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The operation was canceled`

The full stack trace is as follows:

```
Failed during intent acquisition: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The operation was canceled.
---> Microsoft.WindowsAzure.Storage.StorageException: The operation was canceled.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
---> System.Net.Sockets.SocketException (125): Operation canceled
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable`1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)
at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
Request Information
RequestID:
RequestDate:
StatusMessage:
ErrorCode:

--- End of inner exception stack trace ---
at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 140
at DurableTask.AzureStorage.Storage.BlobContainer.ListBlobsAsync(String blobDirectory) in /_/src/DurableTask.AzureStorage/Storage/BlobContainer.cs:line 106
at DurableTask.AzureStorage.Partitioning.BlobLeaseManager.ListLeasesAsync() in /_/src/DurableTask.AzureStorage/Partitioning/BlobLeaseManager.cs:line 79
at DurableTask.AzureStorage.Partitioning.LeaseCollectionBalancer`1.TakeLeasesAsync()
at DurableTask.AzureStorage.Partitioning.LeaseCollectionBalancer`1.LeaseTakerAsync()
```

The source of the error is **also** located [in the AzureStorageClient](https://github.com/Azure/durabletask/blob/619cdbfc62ce6b57d3d1b34e15068eb0e62bec0f/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs#L140). So it could be that the two errors are related.

Again, the `RequestID`, `RequestDate`, `StatusMessage` and `ErrorCode` are empty and the `EventType` that is logged is "PartitionManagerError"

Given the stack trace, and the fact that the event type is "PartitionManagerError", I believe that the source of the error is in the [LeaseCollectionBalancer](https://github.com/Azure/durabletask/blob/d215370465d5494df2a8da679b15f08bd212d834/src/DurableTask.AzureStorage/Partitioning/LeaseCollectionBalancer.cs#L284). If you look, when an exception occurs, then a `PartitionManagerError` is logged and this appears to line up with the stack trace.

My application is using Azure Durable Functions, which I understand uses Azure Blob storage for lease locks. However, it doesn't appear to be surfacing within my code but within something executing with the library (i.e., my code does not receive an exception). It seems to hint that the operation is taking longer than it should.

My Durable Functions are not using Azure Storage for anything aside from the durable function context orchestration (i.e., the code does not directly use a `BlobTrigger`)

What isn't clear:
- Can be resolved via some type of configuration setting. It appears that the default [maximum execution time in the AzureStorageContainer is 2 minutes](https://github.com/Azure/durabletask/blob/516f9990ebb4fd307949e3060d3c1768a10296ec/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs#L28)
- Why the operation was cancelled (as it was not cancelled as a result of the Durable Function execution, AFAIK)