Azure / Azure/azure-functions-host
Function App fails to obtain secret from KeyVault reference, doesn't recover until manually forced to.
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
On December 15th, ~11pm GMT-3, one of my production function apps attempted to authenticate with Azure KeyVault in order to obtain a connection string.
KeyVault replied with an Internal Server Error which, apparently, caused the connection string to never be retrieved, and instead the key vault reference text was being passed as the connection string value.
The function app didn't recover from this, didn't report any error nor try again until ~6 hours later when I noticed, I went to the Configuration settings to validate the references, everything looked good but apparently, that forced another request to KeyVault to retrieve the connection string, which fixed the Function App.
Here's the list of operations performed against my keyvault instance:

The 2 requests at 11:03pm with the warning sign came from this Function App, for 2 different secrets. The one that failed represents the connection string that wasn't set. This Function App didn't refresh the secrets until 5:41am when I manually went to its app settings section in the Azure Portal.
#### Investigative information
Please provide the following:
- Timestamp: ~11pm - ~5:45am GMT-3
- Function App version: ~3
- Function App name: func-loadordersapp-cus-prd
- Function name(s) (as appropriate): GetByScaleTicket, GetById, Patch, Put
- Invocation ID:
- Region: Central US
#### Repro steps
Provide the steps required to reproduce the problem:
I don't know how to reproduce this issue since it seems to have been an internal server error on the KeyVault side during authentication.
#### Expected behavior
The Function App should retry several times to fetch the reference. If possible, it should reuse the last well-known value, or at least fail somehow letting me know it requires some manual intervention.
#### Actual behavior
The environment variable ends up with the literal key vault reference instead of the resolved values. This essentially makes the app unusable until fixed.
#### Known workarounds
Going to the App Settings section in the Azure Portal forces the Function App to request the key vault references again.
#### Related information
* Programming language used: C#, .NET Core 3.1
* Bindings used: HTTP Trigger
Contributor guide
Assessment
This issue has not been assessed yet.