DurableTask should be resilient to Azure Blob storage hangs
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
On our production servers, we have caught an issue in Durable Task framework.
Main issue (how the issue manifests): the call to StopAsync method on TaskHubWorker hangs forever.
I did some deep investigations to the logs and found the original cause. Some facts from logs:
1. The awaiting of "takerTask" in LeaseCollectionBalancer.StopAsync method hangs. This happens because:
2. DownloadLeaseBlob method in BlobLeaseManager.cs hangs.
The latter fact I understood from the logs. We have a logging for all outcoming http requests. During the first 2 days after the service start, every 5 seconds we had the following logs:
- 4 GET requests (to download blob)
- 4 HEAD requests (for fetch attributes)
At some point, the logging stopped (while service was still active for a few days). The last logs contains:
- 4 GET requests
- 3 HEAD requests

All the evidence shows that the hang happens in Azure Blob client. DurableTask SDK should be resilient to it. E.g. it should have a default timeout for all blob operations.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.