Azure / Azure/durabletask

Unforced TaskHubWorker.StopAsync() is extremely slow (not working correctly)

Open
#106 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

The implementation of `TaskHubWorker.StopAsync()` does not appear to be behaving correctly, resulting in extremely slow shutdown sequences.

Here is a minimal repro app:

```csharp
public static void Run()
{
const string HubName = "DurableTaskTestHub";
string servicebusConnectionString = ConfigurationManager.ConnectionStrings["DurableFunctions.ServiceBus"].ConnectionString;
string storageConnectionString = ConfigurationManager.ConnectionStrings["DurableFunctions.Storage"].ConnectionString;

var serviceBusOrchestrationService = new ServiceBusOrchestrationService(
servicebusConnectionString,
HubName,
null,
null,
new ServiceBusOrchestrationServiceSettings());
serviceBusOrchestrationService.CreateIfNotExistsAsync().Await();

Console.WriteLine($"Starting the host.");

var worker = new TaskHubWorker(serviceBusOrchestrationService);
worker.AddTaskOrchestrations(typeof(SimpleOrchestration));
worker.StartAsync().Await();

Console.WriteLine("Stopping host");

Stopwatch sw = Stopwatch.StartNew();
worker.StopAsync().Await();
Console.WriteLine($"Stopped after {sw}.");

Console.WriteLine("Press [ENTER] to exit.");
Console.ReadLine();
}
```

It seems like a bug in the shutdown logic in [WorkItemDispatcher.StopAsync(forced: false)](https://github.com/Azure/durabletask/blob/master/src/DurableTask.Core/WorkItemDispatcher.cs#L163-L168). What I observed while debugging is that `activeFetchers` is always set to 1 even though the system is idle, so we loop and sleep forever until the number of retries expire. The same problem exists when trying to shut down the activity dispatcher.

If I specify `TaskHubWorker.StopAsync(true)`, then the shutdown proceeds quickly.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.