Azure / Azure/durabletask

DT.AzureStorage: Split-brain caused by race condition between ExternalEvent and TaskCompleted

Open
#730 2 comments 0 reactions 1 assignee Claimed by @amdeel View on GitHub
bug dt.azurestorage
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

This issue was discovered by a user of Durable Functions (IcM #299168620 contains the details, including Kusto queries). There appears to be a race condition involving the concurrent processing of the following events:

* ContinueAsNew
* ExternalEvent
* TaskCompleted

What ultimately happens is that two batches of messages are processed separately, one batch containing only external event messages and another containing a task completed message. This particular condition leads to multiple copies of the same orchestration's history being prefetched into memory. The first batch to get processed executes successfully, but the second batch fails with a split-brain warning because the history it prefetched was stale after the first batch updated the history. The message and history prefetch logic is supposed to ensure this never happens, but it appears there may be a race condition where it's still possible.

In terms of impact, sometimes this just creates noise in the logs with very small orchestration processing delays since we need to reprocess the message with a fresh history. However, in other cases, it can cause activities to be executed twice, which is more problematic.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.