Azure / Azure/durabletask

DurableTask should be resilient to Azure Blob storage hangs

Open
#614 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

On our production servers, we have caught an issue in Durable Task framework.

Main issue (how the issue manifests): the call to StopAsync method on TaskHubWorker hangs forever.

I did some deep investigations to the logs and found the original cause. Some facts from logs:
1. The awaiting of "takerTask" in LeaseCollectionBalancer.StopAsync method hangs. This happens because:
2. DownloadLeaseBlob method in BlobLeaseManager.cs hangs.

The latter fact I understood from the logs. We have a logging for all outcoming http requests. During the first 2 days after the service start, every 5 seconds we had the following logs:

- 4 GET requests (to download blob)
- 4 HEAD requests (for fetch attributes)

At some point, the logging stopped (while service was still active for a few days). The last logs contains:

- 4 GET requests
- 3 HEAD requests

![image](https://user-images.githubusercontent.com/55888168/134957638-3982651f-077a-4753-b2e0-3ad852a12afd.png)

All the evidence shows that the hang happens in Azure Blob client. DurableTask SDK should be resilient to it. E.g. it should have a default timeout for all blob operations.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.