Azure / Azure/azure-functions-host
Adding deadlock safety when writing updated MasterKey to the Blob store.
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 37
Description
#### Investigative information
Please provide the following:
Cx information in ICM#254128671
#### Repro steps
The issue is with the Blob Client and the writing of the generated secret to the Blob Store and a deadlock in that code path.
We log the key hashes here, right before persisting keys to the blob repository: SecretManager.cs#L577 which we see in the logs, but we don't see L579 returning (and we don't even get an exception). This points to a bug in the BlobStorageSecretsRepository.cs#L114 and any exception not returning.
```
2021-07-21T08:24:32.2367212Z
Microsoft.Azure.WebJobs.Script.WebHost.SecretManager
Encryption keys hashes: MACHINEKEY_DecryptionKey=QbPnTh7Cg5ZSXLc6xFmv5h6yZDHwLxtqXM4tUFTbvlY=;
```
#### Expected behavior
Provide a description of the expected behavior.
#### Actual behavior
The customer tried changing the _master key by clicking the "Renew key value" button in the portal at 2021-07-21T08:24:43 UTC, but that attempt failed (it returned a 400). Since the change operation failed, Cx assumed the key value did not change.
But after some time (8:37ish), all the subsequent calls started coming out as 401s and thus failing.
This needs to be investigated and fixed.
#### Known workarounds
Cx tried to change the _master key on 2021-07-26T05:04:48 UTC again which succeeded and resolved the 401 issues.
#### Related information
More info in ICM.
Contributor guide
Research direction
Start with SecretManager.cs around L577-L579 and BlobStorageSecretsRepository.cs around L114, using the provided logs and renewal timeline to trace the write path. Reproduce or test the master-key renewal path and identify whether it can deadlock without returning an exception. Done means the update completes safely and subsequent requests do not fail with 401 responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100