Azure / Azure/durabletask

DT.AzureStorage: Split-brain caused by race condition between ExternalEvent and TaskCompleted

Aperta
#730 2 commenti 0 reazioni 1 assegnatario Rivendicata da @amdeel Vedi su GitHub
bug dt.azurestorage
Lingua principale
C#
Stelle
1.7k
Fork
335
Merge medio
2g 23h
PR unite (30g)
6

Descrizione

This issue was discovered by a user of Durable Functions (IcM #299168620 contains the details, including Kusto queries). There appears to be a race condition involving the concurrent processing of the following events:

* ContinueAsNew
* ExternalEvent
* TaskCompleted

What ultimately happens is that two batches of messages are processed separately, one batch containing only external event messages and another containing a task completed message. This particular condition leads to multiple copies of the same orchestration's history being prefetched into memory. The first batch to get processed executes successfully, but the second batch fails with a split-brain warning because the history it prefetched was stale after the first batch updated the history. The message and history prefetch logic is supposed to ensure this never happens, but it appears there may be a race condition where it's still possible.

In terms of impact, sometimes this just creates noise in the logs with very small orchestration processing delays since we need to reprocess the message with a fresh history. However, in other cases, it can cause activities to be executed twice, which is more problematic.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.