Azure / Azure/azure-functions-host

"Azure functions runtime is unreachable" error exactly one year after app deployment.

Open
#9,113 10 comments 2 reactions 0 assignees View on GitHub
supportability
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

### The sequence of events

1. Exactly a year ago (on 2/23/202**2**), I'd deployed my APIs to my Azure function app.

![image](https://user-images.githubusercontent.com/2327905/221223124-e3b947de-76bd-4ed3-a763-287249e2d775.png)

2. Yesterday (i.e. a year later, on 2/23/202**3**), I noticed that the APIs started returning `503 / unavailable` errors. Upon logging into the Azure portal I noticed the `Azure functions runtime is unreachable` error. Restarting the app didn't help.

![image](https://user-images.githubusercontent.com/2327905/221225083-b5b47128-8cd2-4544-950a-7437a8bf2f51.png)

3. The `Diagnose and solve problems` tab on Azure portal led to the following discovery: 11 instances of `Microsoft.AspNetCore.Connections.ConnectionAbortedException`. Not entirely sure how much this is related to the app's downtime.

![Screenshot 2023-02-23 234711](https://user-images.githubusercontent.com/2327905/221226561-3bbc8db4-2298-48b2-b028-88c98317f56a.png)

![Screenshot 2023-02-23 234800](https://user-images.githubusercontent.com/2327905/221226603-a76752b9-81e6-48e8-be70-e90e1641aba4.png)

4. I opened a support ticket `2302230030002367` for this issue. While the root-cause investigation was inconclusive, I was able to resolve the issue by simply redeploying the app once again.

### Probable root cause

1. I later stumbled upon the possible root cause (thanks to [Erik_ERBBQ](https://twitter.com/Erik_ERBBQ/status/1629092771519684612)). The value of `WEBSITE_RUN_FROM_PACKAGE` app setting is a SAS token that expires exactly one year after deployment!

![image](https://user-images.githubusercontent.com/2327905/221238860-8c4d723b-18f9-4d9e-960c-de0d63b2c3b0.png)

![image](https://user-images.githubusercontent.com/2327905/221239457-ab45b152-7a9c-48d4-9bb4-2da3d1fa2a9f.png)

2. The SAS token is generated by the Azure pipeline deployment task `AzureFunctionApp@1` which I use to deploy my app.

![image](https://user-images.githubusercontent.com/2327905/221244741-08f57738-9b4a-4900-81c2-3936bb472839.png)

3. And here is the line of code to blame ([LINK](https://github.com/microsoft/azure-pipelines-tasks/blob/30c99c8a7013d012f14c129e8ee212c7c622008a/Tasks/AzureFunctionAppV1/deploymentProvider/ConsumptionWebAppDeploymentProvider.ts#L90)). So I guess, this bug really belongs in that github repo(?)

![image](https://user-images.githubusercontent.com/2327905/221245176-46870a89-dfae-4969-a523-5de3f868a1d1.png)

#### Investigative information

- SubscriptionID: `014a6441-97ae-45e2-8b37-ff577abb1086`
- Function App version: `4x`
- Function App name: `cloudskewfunctionsprod`
- Function name(s) (as appropriate): multiple
- Invocation ID: N/A
- Region: `west europe`
- Timestamp: Based on the metric chart below, the outage started at 11:17 AM on 2/23/2023 UTC time. This is corroborated by the data from the `Diagnose and solve problems` portal tab. The outage finally ended when I redeployed the app around 6:15 PM.

![image](https://user-images.githubusercontent.com/2327905/221230143-31926767-3147-45b1-a844-21f8142c2500.png)

#### Related information

* Programming language used: C# with .NET 6
* Links to source: Unfortunately this is a private github repo.
* Bindings used: The APIs are all HTTP-triggered. No input or output bindings are used (I mostly use the SDK for all IO). A timer-triggered function also exists in the same app.
* App settings used: See screenshot below

![image](https://user-images.githubusercontent.com/2327905/221229308-b6bc8970-2b61-4456-b016-30cdb1a854ee.png)

#### Other investigation notes

1. I did look through the MSDN documentation for the `Azure functions runtime is unreachable` error ([LINK](https://learn.microsoft.com/en-us/azure/azure-functions/functions-recover-storage-account)). But nothing conclusive stood out.
2. Also rotated the storage account key/connectionString used in the `AzureWebJobsStorage` app configuration setting. It didn't help.
3. The following github issues might be related (but I'm not 100% sure).
* https://github.com/Azure/azure-functions-host/issues?q=is%3Aissue+unreachable
* https://github.com/Azure/azure-functions/issues?q=is%3Aissue+unreachable

Contributor guide

Open the contributing guide

Research direction

Start with the linked AzureFunctionApp@1 source, Tasks/AzureFunctionAppV1/deploymentProvider/ConsumptionWebAppDeploymentProvider.ts around line 90, and compare how WEBSITE_RUN_FROM_PACKAGE is generated with the reported one-year outage. Confirm whether the behavior belongs in that deployment task or azure-functions-host, then document a reproducible failure and the appropriate fix location.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, typescript
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.