Azure / Azure/Azure-Functions

Unreliable zip deployment to the staging slot.

Open
#1,800 2 comments 0 reactions 1 assignee Claimed by @amamounelsayed View on GitHub
needs-investigation
Dominant language
PowerShell
Stars
1.1k
Forks
215
Avg merge
4h 2m
Merged PRs (30d)
1

Description

Hi there,

We are experiencing an intermediate issue when deploying our function app using Azure CLI as a zip package to the **staging** slot (See errors below).

Our function is configured to use:

1. Consumption plan.
2. Node.js 12
3. West US
4. Slots: staging, production

Additional tools:

azure-cli 2.16.0
core 2.16.0
telemetry 1.0.6

Your CLI is up-to-date.

### Here is the automated build process:

1. Update app settings in the **staging** slot

```
az webapp config appsettings set --slot staging --resource-group zzzResource --name zzz-test-us-dev --settings --verbose
```

2. Deploy to the **staging** slot

```
az functionapp deployment source config-zip --resource-group zzzResource --name zzz-test-us-dev --src --slot staging --verbose
```

3. Pinging **staging** slot instance to keep it warm for a minute.

4. Executing slot swap

```
az functionapp deployment slot swap --action swap --resource-group zzzResource --name zzz-test-us-dev --slot staging --target-slot production --verbose
```

### Additional app settings:

WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG=1
WEBSITE_RUN_FROM_PACKAGE=1

This works most of the time, but occasionally, the deployment to the staging slot fails, and it ends up taking down the production slot with it.

### Errors:

```
Deploying "zzz-test-us-dev" to the "US" region.
Getting 'zzz-test-us-dev' app settings
Updating app version to 3
Done zipping the "zzz-test-us-dev" function!

Updating 'zzz-test-us-dev' app settings
Command ran in 1.837 seconds (init: 0.061, invoke: 1.776)

Deploying Azure function 'zzz-test-us-dev.'

Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
Configuring default logging for the app, if not already enabled

Zip deployment failed. {'id': '6b96f759730146e09f4afb809c420b87', 'status': 3, 'status_text': '', 'author_email': 'N/A', 'author': 'N/A', 'deployer': 'ZipDeploy', 'message': 'Created via a push deployment', 'progress': '', 'received_time': '2020-12-14T20:08:34.2407885Z', 'start_time': '2020-12-14T20:08:34.2407885Z', 'end_time': '2020-12-14T20:08:34.8501901Z', 'last_success_end_time': None, 'complete': True, 'active': False, 'is_temp': False, 'is_readonly': True, 'url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest', 'log_url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest/log', 'site_name': 'zzz-test-us-dev', 'provisioningState': None}. Please run the command az webapp log deployment show -n zzz-test-us-dev -g zzzResource
Command ran in 17.114 seconds (init: 0.066, invoke: 17.048)

Unable to deploy the function app to the 'staging' slot. Error: Error: Command failed: az functionapp deployment source config-zip --resource-group zzzResource --name zzz-test-us-dev --src ./dist/zipPackages/zzz-test-us-dev.zip --slot staging --verbose

Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
Configuring default logging for the app, if not already enabled

Zip deployment failed. {'id': '6b96f759730146e09f4afb809c420b87', 'status': 3, 'status_text': '', 'author_email': 'N/A', 'author': 'N/A', 'deployer': 'ZipDeploy', 'message': 'Created via a push deployment', 'progress': '', 'received_time': '2020-12-14T20:08:34.2407885Z', 'start_time': '2020-12-14T20:08:34.2407885Z', 'end_time': '2020-12-14T20:08:34.8501901Z', 'last_success_end_time': None, 'complete': True, 'active': False, 'is_temp': False, 'is_readonly': True, 'url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest', 'log_url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest/log', 'site_name': 'zzz-test-us-dev', 'provisioningState': None}. Please run the command az webapp log deployment show -n zzz-test-us-dev -g zzzResource
Command ran in 17.114 seconds (init: 0.066, invoke: 17.048)

```
Why would zip file deployment fail? The same zip file will get successfully deployed on a second or third attempt (IMHO: Not a zip package issue). In addition, there is no useful information in the error message.

We have seen this error quite a bit lately.

### **Production** instance ping results captured during the deployment to the **staging** slot:
```
>>> App Version <<< : 2 [Released On: 12/14/2020] - (76 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (80 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (77 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (89 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (96 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (81 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (450 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (102 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (356 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (85 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (107 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (69 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (85 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (77 ms)
ERROR: 503 - The service is unavailable. - (90 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (6043 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (104 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (74 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (92 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (106 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (73 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (70 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (98 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (97 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (100 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (89 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (91 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (85 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (89 ms)
```

How can we avoid failed deployment to **staging** slot bringing the **production** instance down (See the production instance ping results above with 503 error)?

Any help will be appreciated.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.