Detect orchestrations that get into a bad state
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
[Per the bug in this PR](https://github.com/Azure/durabletask/pull/480), we have gotten into states where an orchestration is scheduled but never actually starts. We would like to have better visibility and recoverability when this happens.
As a workaround for detection, we have created a timer trigger that uses the `durableClient.ListInstancesAsync` method to find orchestrations with the `OrchestrationRuntimeStatus.Pending` status that have a `CreatedTime` that is older than 5 minutes. While I know this could falsely flag orchestrations that were scheduled but never executed because machines were down, it does give us some visibility that something is wrong before a customer complains.
Further exacerbating the problem, if the stalled orchestration is a singleton, there is no graceful recovery. Terminating and starting the orchestration again doesn't do anything to get it back into a usable state.
@davidmrdavid
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.