Azure / Azure/azure-functions-host

Add retries for BlobStorageSecretsRepository precondition failures (concurrency scenarios)

Open
#8,835 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

Our blob storage persistence code can fail with 412 Precondition errors if two separate updates are racing to add keys to the same file. For example if two concurrent requests are made to add/modify host keys, one update will succeed the other will fail. This is due to the precondition code [here](https://github.com/Azure/azure-functions-host/blob/5487571e30a5c01b77ee306abe6ada45a7dd2567/src/WebJobs.Script.WebHost/Security/KeyManagement/BlobStorageSecretsRepository.cs#L181) which ensures that no intervening updates happen when writing the blob. Same issue likely exists for the file based provider, but we should focus on blob since it's the default.

We should consider putting retries in place for this, so the host will re-read the current state of the blob as updated by another request, and use the new Etag, allowing the update to succeed. Perhaps we do up to three retries.

Contributor guide

Open the contributing guide

Research direction

Start in src/WebJobs.Script.WebHost/Security/KeyManagement/BlobStorageSecretsRepository.cs at the precondition code around line 181. Trace how the blob is read and written when concurrent updates produce a 412 Precondition error, then verify that the repository can retry using the latest blob state and Etag, with completion shown by concurrent updates succeeding after the proposed retry limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.