Azure / Azure/durabletask

Detect orchestrations that get into a bad state

Open
#483 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

[Per the bug in this PR](https://github.com/Azure/durabletask/pull/480), we have gotten into states where an orchestration is scheduled but never actually starts. We would like to have better visibility and recoverability when this happens.

As a workaround for detection, we have created a timer trigger that uses the `durableClient.ListInstancesAsync` method to find orchestrations with the `OrchestrationRuntimeStatus.Pending` status that have a `CreatedTime` that is older than 5 minutes. While I know this could falsely flag orchestrations that were scheduled but never executed because machines were down, it does give us some visibility that something is wrong before a customer complains.

Further exacerbating the problem, if the stalled orchestration is a singleton, there is no graceful recovery. Terminating and starting the orchestration again doesn't do anything to get it back into a usable state.

@davidmrdavid

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.