Azure / Azure/azure-functions-host
Add retries for BlobStorageSecretsRepository precondition failures (concurrency scenarios)
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
Our blob storage persistence code can fail with 412 Precondition errors if two separate updates are racing to add keys to the same file. For example if two concurrent requests are made to add/modify host keys, one update will succeed the other will fail. This is due to the precondition code [here](https://github.com/Azure/azure-functions-host/blob/5487571e30a5c01b77ee306abe6ada45a7dd2567/src/WebJobs.Script.WebHost/Security/KeyManagement/BlobStorageSecretsRepository.cs#L181) which ensures that no intervening updates happen when writing the blob. Same issue likely exists for the file based provider, but we should focus on blob since it's the default.
We should consider putting retries in place for this, so the host will re-read the current state of the blob as updated by another request, and use the new Etag, allowing the update to succeed. Perhaps we do up to three retries.
Contributor guide
Research direction
Start in src/WebJobs.Script.WebHost/Security/KeyManagement/BlobStorageSecretsRepository.cs at the precondition code around line 181. Trace how the blob is read and written when concurrent updates produce a 412 Precondition error, then verify that the repository can retry using the latest blob state and Etag, with completion shown by concurrent updates succeeding after the proposed retry limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100