DT.AzureStorage: Split-brain caused by race condition between ExternalEvent and TaskCompleted
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
This issue was discovered by a user of Durable Functions (IcM #299168620 contains the details, including Kusto queries). There appears to be a race condition involving the concurrent processing of the following events:
* ContinueAsNew
* ExternalEvent
* TaskCompleted
What ultimately happens is that two batches of messages are processed separately, one batch containing only external event messages and another containing a task completed message. This particular condition leads to multiple copies of the same orchestration's history being prefetched into memory. The first batch to get processed executes successfully, but the second batch fails with a split-brain warning because the history it prefetched was stale after the first batch updated the history. The message and history prefetch logic is supposed to ensure this never happens, but it appears there may be a race condition where it's still possible.
In terms of impact, sometimes this just creates noise in the logs with very small orchestration processing delays since we need to reprocess the message with a fresh history. However, in other cases, it can cause activities to be executed twice, which is more problematic.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.