Azure / Azure/azure-functions-host

Function App fails to obtain secret from KeyVault reference, doesn't recover until manually forced to.

Open
#8,009 2 comments 0 reactions 1 assignee Assigned to @soninaren View on GitHub
needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

On December 15th, ~11pm GMT-3, one of my production function apps attempted to authenticate with Azure KeyVault in order to obtain a connection string.
KeyVault replied with an Internal Server Error which, apparently, caused the connection string to never be retrieved, and instead the key vault reference text was being passed as the connection string value.

The function app didn't recover from this, didn't report any error nor try again until ~6 hours later when I noticed, I went to the Configuration settings to validate the references, everything looked good but apparently, that forced another request to KeyVault to retrieve the connection string, which fixed the Function App.

Here's the list of operations performed against my keyvault instance:
![image](https://user-images.githubusercontent.com/22510674/146344656-eda5713b-bf15-4b0f-9d78-1ea6e64375d0.png)
The 2 requests at 11:03pm with the warning sign came from this Function App, for 2 different secrets. The one that failed represents the connection string that wasn't set. This Function App didn't refresh the secrets until 5:41am when I manually went to its app settings section in the Azure Portal.

#### Investigative information

Please provide the following:

- Timestamp: ~11pm - ~5:45am GMT-3
- Function App version: ~3
- Function App name: func-loadordersapp-cus-prd
- Function name(s) (as appropriate): GetByScaleTicket, GetById, Patch, Put
- Invocation ID:
- Region: Central US

#### Repro steps

Provide the steps required to reproduce the problem:

I don't know how to reproduce this issue since it seems to have been an internal server error on the KeyVault side during authentication.

#### Expected behavior

The Function App should retry several times to fetch the reference. If possible, it should reuse the last well-known value, or at least fail somehow letting me know it requires some manual intervention.

#### Actual behavior

The environment variable ends up with the literal key vault reference instead of the resolved values. This essentially makes the app unusable until fixed.

#### Known workarounds

Going to the App Settings section in the Azure Portal forces the Function App to request the key vault references again.

#### Related information

* Programming language used: C#, .NET Core 3.1
* Bindings used: HTTP Trigger

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.