Azure / Azure/azure-functions-host

Adding deadlock safety when writing updated MasterKey to the Blob store.

Open
#7,635 1 comment 0 reactions 0 assignees View on GitHub
needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 11h
Merged PRs (30d)
37

Description

#### Investigative information

Please provide the following:
Cx information in ICM#254128671

#### Repro steps

The issue is with the Blob Client and the writing of the generated secret to the Blob Store and a deadlock in that code path.

We log the key hashes here, right before persisting keys to the blob repository: SecretManager.cs#L577 which we see in the logs, but we don't see L579 returning (and we don't even get an exception). This points to a bug in the BlobStorageSecretsRepository.cs#L114 and any exception not returning.
```
2021-07-21T08:24:32.2367212Z
Microsoft.Azure.WebJobs.Script.WebHost.SecretManager
Encryption keys hashes: MACHINEKEY_DecryptionKey=QbPnTh7Cg5ZSXLc6xFmv5h6yZDHwLxtqXM4tUFTbvlY=;
```

#### Expected behavior

Provide a description of the expected behavior.

#### Actual behavior

The customer tried changing the _master key by clicking the "Renew key value" button in the portal at 2021-07-21T08:24:43 UTC, but that attempt failed (it returned a 400). Since the change operation failed, Cx assumed the key value did not change.

But after some time (8:37ish), all the subsequent calls started coming out as 401s and thus failing.

This needs to be investigated and fixed.

#### Known workarounds

Cx tried to change the _master key on 2021-07-26T05:04:48 UTC again which succeeded and resolved the 401 issues.

#### Related information

More info in ICM.

Contributor guide

Open the contributing guide

Research direction

Start with SecretManager.cs around L577-L579 and BlobStorageSecretsRepository.cs around L114, using the provided logs and renewal timeline to trace the write path. Reproduce or test the master-key renewal path and identify whether it can deadlock without returning an exception. Done means the update completes safely and subsequent requests do not fail with 401 responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.