Azure / Azure/Azure-Functions

Logic App (standard) deployment fails intermittently - kudu service unavailable 503

Open
#2,488 4 comments 7 reactions 0 assignees View on GitHub
bug
Dominant language
PowerShell
Stars
1.1k
Forks
215
Avg merge
4h 2m
Merged PRs (30d)
1

Description

When deploying a Logic App (standard) via DevOps pipelines and the resource doesn't exist, approx every third deployment will fail with a "kudu service unavailable 503" error.

The deployment is done in two steps:

Infrastructure
ARM template is deployed using the AzureResourceManagerTemplateDeployment@3 task.
The deployment is successful, however:
- the App Service overview page displays "error" instead of the runtime version
- kudu becomes unresponsive
- the Log Stream is unavailable
- In the activity log in the azure portal, there is a Validate action which fails with a message: “The gateway did not receive a response from ‘Microsoft.Web’ within the specified time period: 504"

Stopping/starting the app or changing the App Service Plan fixes this, and the ASP CPU/memory usage is near 0%, which seems to point to an unhandled runtime exception.

In the pipeline, if also deploying the application using AzureFunctionApp@1 and the App Service is in the error state:
- Within a few minutes Azure DevOps displays “No message found for this issue”
- It may take up to 45 mins to deploy, or it may fail with the output below:

```
Starting: Deploying Standard Logic App

Task : Azure Functions Deploy
Description : Update a function app with .NET, Python, JavaScript, PowerShell, Java based web applications
Version : 2.238.1
Author : Microsoft Corporation
Help : https://aka.ms/azurefunctiontroubleshooting

Got service connection details for Azure App Service:'logicapp-dev'
NOTE: Function app is VNet integrated.
Deleting App Service Application settings. Data: ["WEBSITE_RUN_FROM_ZIP","WEBSITE_RUN_FROM_PACKAGE"]
App Service Application settings are already present.
Validating deployment package for functions app before Zip Deploy
Package deployment using ZIP Deploy initiated.
##[error]Failed to deploy web package to App Service.
##[warning]Can't find loc string for key: KuduStackTraceURL
##[error]KuduStackTraceURL https://$logicapp-dev:***@logicapp-dev.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
##[error]Error: Error: Failed to deploy web package to App Service. Service Unavailable (CODE: 503)
##[warning]Error: Failed to update deployment history. Error: Service Unavailable (CODE: 503)
App Service Application URL: https://logicapp-dev.azurewebsites.net
Finishing: Deploying Standard Logic App
```

Following the link to the kudu trace logs provides the following:

```
{
"Message": "An error has occurred.",
"ExceptionMessage": "Illegal characters in path.",
"ExceptionType": "System.ArgumentException",
"StackTrace": " at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)\r\n
at System.IO.Path.Combine(String path1, String path2)\r\n
at Kudu.Services.Infrastructure.VfsControllerBase.GetOriginalLocalFilePath()
in C:\\__w\\1\\s\\Kudu.Services\\Infrastructure\\VfsControllerBase.cs:line 327\r\n
at Kudu.Services.Infrastructure.VfsControllerBase.GetLocalFilePath()
in C:\\__w\\1\\s\\Kudu.Services\\Infrastructure\\VfsControllerBase.cs:line 316\r\n
at Kudu.Services.Infrastructure.VfsControllerBase.GetItem()
in C:\\__w\\1\\s\\Kudu.Services\\Infrastructure\\VfsControllerBase.cs:line 52\r\n
at lambda_method(Closure , Object , Object[] )\r\n
at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor

...continued...
}
```

I've spent about 5 days ruling out the following:
- Application itself (workflows, connections, parameters, host.json etc)
- API connections
- App Service Plan (tried deleting + scaling up)
- Application runtime (tried reverting from dotnet to Node)
- ARM template (new logic app ARM template created in Azure Portal also fails)
- Pipeline task (tried using AzureFunctionApp@2)
- Logic App name (tried renaming the logic app and fileshare, initially succeeded in DEV/SIT then same error in UAT)
- Storage Account Firewall (tried enabling public access)
- Not deploying file share manually (deploying a Standard Logic App also provisions one automatically unless WEBSITE_SKIP_CONTENTSHARE_VALIDATION is set to 1)
- Deleting deployments folder via Kudu (some users reported the residual files being there related to the connection between DevOps and Azure)

Notes:
- Logic App is VNet integrated using a storage account in a separate resource group.
- Storage account is configured with private endpoints and to allow access to Trusted Azure Services.
- App Settings have WEBSITE_CONTENTAZUREFILECONNECTIONSTRING, WEBSITE_CONTENTOVERVNET and WEBSITE_SKIP_CONTENTSHARE_VALIDATION configured (storage File Share is deployed first)
- App Insights show no failures from the logic app to the storage account file share
- Region is Australia Southeast

Can you please provide any other potential causes of the intermittent issue or triage as required.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.