Azure / Azure/azure-cli

Azure function: sporadic and unpredictable failures during code deployment

Open
#32,551 5 comments 0 reactions 4 assignees Claimed by @kamperiadis View on GitHub
act-observability-squad Auto-Assign bug customer-reported Functions Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

Hello. We are trying to use Azure functions in our project.

I created a pipeline with a script that creates a zip file (size is approximately 10 MB), creates an Azure function, and deploys a zip file into it.

We are running this script using Azure pipelines on an Azure machine. I mean, we are not using self-hosted solutions for this project. So I am sure that zip file is flying from Azure machine to Azure machine in the Azure internal network. Please don't suspect a network issue.

The version of the `az` tool is the latest possible because we are running it on the Azure DevOps machine as a part of the pipeline script.

We have received an issue in the simplest and most reliable part of the script:

```
echo 'Deploy application'
az functionapp deployment source config-zip \
--name '$(FUNCTION_NAME)' \
--resource-group '$(AZ_RG)' \
--src "$APP_PATH"
```

The failing output is the following:
```
Deploy application
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
WARNING: Deployment endpoint responded with status code 202 for deployment id "***"
ERROR: Failed to connect to 'https://***.scm.azurewebsites.net/api/deployments/***/log' with status code '400' and reason 'Bad Request'
```

I made about 50 deployments and received about 3 failed deployments because of this error. So the **failure rate** for deploying Azure functions is **about 6%**, which is too high!

For now I made a deployment for development purposes only and asked our developers to rerun the pipeline when it fails because of an Azure issue. But tomorrow I will need to make a production deployment, and the customer will ask me a question: why does the pipeline fail?

Can you please provide me a good reason why the chance of successful deployment to Azure Function is so low? So I will resend it to the customer. Because I need to make things clear: this is not my fault; this is Azure's fault. Thank you.

### Related command

```
echo 'Deploy application'
az functionapp deployment source config-zip \
--name '$(FUNCTION_NAME)' \
--resource-group '$(AZ_RG)' \
--src "$APP_PATH"
```

### Errors

```
Deploy application
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
WARNING: Deployment endpoint responded with status code 202 for deployment id "***"
ERROR: Failed to connect to 'https://***.scm.azurewebsites.net/api/deployments/***/log' with status code '400' and reason 'Bad Request'
```

### Issue script & Debug output

Full script:

```
stages:
- stage: DeployFunction
displayName: 'Deploy Function'
jobs:
- job: DeployFunction
displayName: 'Deploy Function'
steps:
- task: AzureCLI@2
displayName: 'Deploy Function'
inputs:
azureSubscription: '$(AZ_SUB)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
set -euo pipefail

...

echo 'Create application zip'
APP_PATH='$(Agent.TempDirectory)/.application.zip'
...

echo 'Create app insights'
az monitor app-insights component create \
--app '$(APP_INSIGHTS_NAME)' \
--location '$(AZ_LOC)' \
--resource-group '$(AZ_RG)' \
--application-type 'web'

if ! az functionapp show --name '$(FUNCTION_NAME)' --resource-group '$(AZ_RG)' >/dev/null 2>&1; then
echo 'Create function'
az functionapp create \
--name '$(FUNCTION_NAME)' \
--resource-group '$(AZ_RG)' \
--storage-account '$(AZ_STO)' \
--flexconsumption-location '$(AZ_LOC)' \
--app-insights '$(APP_INSIGHTS_NAME)' \
--functions-version '$(FUNCTION_VERSION)' \
--runtime '$(RUNTIME_NAME)' \
--runtime-version '$(RUNTIME_VERSION)' \
--os-type 'Linux'
fi

echo 'Set application env'
az functionapp config appsettings set \
--name '$(FUNCTION_NAME)' \
--resource-group '$(AZ_RG)' \
--settings \
...

echo 'Deploy application'
az functionapp deployment source config-zip \
--name '$(FUNCTION_NAME)' \
--resource-group '$(AZ_RG)' \
--src "$APP_PATH"

echo 'Show application info'
az functionapp function list \
--name '$(FUNCTION_NAME)' \
--resource-group '$(AZ_RG)' \
-o 'table'
```

### Expected behavior

The error may appear, but a failure rate of 6% is too high.

### Environment Summary

`azure-cli`: latest version

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.