Azure / Azure/durabletask

DT.AzureStorage: wrong blob may win when split-brain instances race

Open
#890 0 comments 6 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

In split-brain situations, two hosts may try to write the same instance history simultaneously.

To protect against state corruption, the `AzureTableTrackingStore` uses an e-Tag, stored in a sentinel row, which ensures that only one of the racing accesses can actually update the table. The losing access will then abandon the orchestration work item, thus resolving the split-brain situation (only the winner remains active).

Unfortunately, this mechanism is not sufficient to also protect against *races on blobs* that can happen when table properties are too large and are stored in a blob. In that case, the losing access (in terms of e-tag) may still be able to overwrite the blob of the winning access, leading to incorrect results as reported in https://github.com/Azure/azure-functions-durable-extension/issues/2437.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.