Azure / Azure/azure-functions-host
"Azure functions runtime is unreachable" error exactly one year after app deployment.
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
### The sequence of events
1. Exactly a year ago (on 2/23/202**2**), I'd deployed my APIs to my Azure function app.

2. Yesterday (i.e. a year later, on 2/23/202**3**), I noticed that the APIs started returning `503 / unavailable` errors. Upon logging into the Azure portal I noticed the `Azure functions runtime is unreachable` error. Restarting the app didn't help.

3. The `Diagnose and solve problems` tab on Azure portal led to the following discovery: 11 instances of `Microsoft.AspNetCore.Connections.ConnectionAbortedException`. Not entirely sure how much this is related to the app's downtime.


4. I opened a support ticket `2302230030002367` for this issue. While the root-cause investigation was inconclusive, I was able to resolve the issue by simply redeploying the app once again.
### Probable root cause
1. I later stumbled upon the possible root cause (thanks to [Erik_ERBBQ](https://twitter.com/Erik_ERBBQ/status/1629092771519684612)). The value of `WEBSITE_RUN_FROM_PACKAGE` app setting is a SAS token that expires exactly one year after deployment!


2. The SAS token is generated by the Azure pipeline deployment task `AzureFunctionApp@1` which I use to deploy my app.

3. And here is the line of code to blame ([LINK](https://github.com/microsoft/azure-pipelines-tasks/blob/30c99c8a7013d012f14c129e8ee212c7c622008a/Tasks/AzureFunctionAppV1/deploymentProvider/ConsumptionWebAppDeploymentProvider.ts#L90)). So I guess, this bug really belongs in that github repo(?)

#### Investigative information
- SubscriptionID: `014a6441-97ae-45e2-8b37-ff577abb1086`
- Function App version: `4x`
- Function App name: `cloudskewfunctionsprod`
- Function name(s) (as appropriate): multiple
- Invocation ID: N/A
- Region: `west europe`
- Timestamp: Based on the metric chart below, the outage started at 11:17 AM on 2/23/2023 UTC time. This is corroborated by the data from the `Diagnose and solve problems` portal tab. The outage finally ended when I redeployed the app around 6:15 PM.

#### Related information
* Programming language used: C# with .NET 6
* Links to source: Unfortunately this is a private github repo.
* Bindings used: The APIs are all HTTP-triggered. No input or output bindings are used (I mostly use the SDK for all IO). A timer-triggered function also exists in the same app.
* App settings used: See screenshot below

#### Other investigation notes
1. I did look through the MSDN documentation for the `Azure functions runtime is unreachable` error ([LINK](https://learn.microsoft.com/en-us/azure/azure-functions/functions-recover-storage-account)). But nothing conclusive stood out.
2. Also rotated the storage account key/connectionString used in the `AzureWebJobsStorage` app configuration setting. It didn't help.
3. The following github issues might be related (but I'm not 100% sure).
* https://github.com/Azure/azure-functions-host/issues?q=is%3Aissue+unreachable
* https://github.com/Azure/azure-functions/issues?q=is%3Aissue+unreachable
Contributor guide
Research direction
Start with the linked AzureFunctionApp@1 source, Tasks/AzureFunctionAppV1/deploymentProvider/ConsumptionWebAppDeploymentProvider.ts around line 90, and compare how WEBSITE_RUN_FROM_PACKAGE is generated with the reported one-year outage. Confirm whether the behavior belongs in that deployment task or azure-functions-host, then document a reproducible failure and the appropriate fix location.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, typescript
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100