Unreliable zip deployment to the staging slot.
- Dominant language
- PowerShell
- Stars
- 1.1k
- Forks
- 215
- Avg merge
- 4h 2m
- Merged PRs (30d)
- 1
Description
Hi there,
We are experiencing an intermediate issue when deploying our function app using Azure CLI as a zip package to the **staging** slot (See errors below).
Our function is configured to use:
1. Consumption plan.
2. Node.js 12
3. West US
4. Slots: staging, production
Additional tools:
azure-cli 2.16.0
core 2.16.0
telemetry 1.0.6
Your CLI is up-to-date.
### Here is the automated build process:
1. Update app settings in the **staging** slot
```
az webapp config appsettings set --slot staging --resource-group zzzResource --name zzz-test-us-dev --settings --verbose
```
2. Deploy to the **staging** slot
```
az functionapp deployment source config-zip --resource-group zzzResource --name zzz-test-us-dev --src --slot staging --verbose
```
3. Pinging **staging** slot instance to keep it warm for a minute.
4. Executing slot swap
```
az functionapp deployment slot swap --action swap --resource-group zzzResource --name zzz-test-us-dev --slot staging --target-slot production --verbose
```
### Additional app settings:
WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG=1
WEBSITE_RUN_FROM_PACKAGE=1
This works most of the time, but occasionally, the deployment to the staging slot fails, and it ends up taking down the production slot with it.
### Errors:
```
Deploying "zzz-test-us-dev" to the "US" region.
Getting 'zzz-test-us-dev' app settings
Updating app version to 3
Done zipping the "zzz-test-us-dev" function!
Updating 'zzz-test-us-dev' app settings
Command ran in 1.837 seconds (init: 0.061, invoke: 1.776)
Deploying Azure function 'zzz-test-us-dev.'
Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
Configuring default logging for the app, if not already enabled
Zip deployment failed. {'id': '6b96f759730146e09f4afb809c420b87', 'status': 3, 'status_text': '', 'author_email': 'N/A', 'author': 'N/A', 'deployer': 'ZipDeploy', 'message': 'Created via a push deployment', 'progress': '', 'received_time': '2020-12-14T20:08:34.2407885Z', 'start_time': '2020-12-14T20:08:34.2407885Z', 'end_time': '2020-12-14T20:08:34.8501901Z', 'last_success_end_time': None, 'complete': True, 'active': False, 'is_temp': False, 'is_readonly': True, 'url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest', 'log_url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest/log', 'site_name': 'zzz-test-us-dev', 'provisioningState': None}. Please run the command az webapp log deployment show -n zzz-test-us-dev -g zzzResource
Command ran in 17.114 seconds (init: 0.066, invoke: 17.048)
Unable to deploy the function app to the 'staging' slot. Error: Error: Command failed: az functionapp deployment source config-zip --resource-group zzzResource --name zzz-test-us-dev --src ./dist/zipPackages/zzz-test-us-dev.zip --slot staging --verbose
Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
Configuring default logging for the app, if not already enabled
Zip deployment failed. {'id': '6b96f759730146e09f4afb809c420b87', 'status': 3, 'status_text': '', 'author_email': 'N/A', 'author': 'N/A', 'deployer': 'ZipDeploy', 'message': 'Created via a push deployment', 'progress': '', 'received_time': '2020-12-14T20:08:34.2407885Z', 'start_time': '2020-12-14T20:08:34.2407885Z', 'end_time': '2020-12-14T20:08:34.8501901Z', 'last_success_end_time': None, 'complete': True, 'active': False, 'is_temp': False, 'is_readonly': True, 'url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest', 'log_url': 'https://zzz-test-us-dev-staging.scm.azurewebsites.net/api/deployments/latest/log', 'site_name': 'zzz-test-us-dev', 'provisioningState': None}. Please run the command az webapp log deployment show -n zzz-test-us-dev -g zzzResource
Command ran in 17.114 seconds (init: 0.066, invoke: 17.048)
```
Why would zip file deployment fail? The same zip file will get successfully deployed on a second or third attempt (IMHO: Not a zip package issue). In addition, there is no useful information in the error message.
We have seen this error quite a bit lately.
### **Production** instance ping results captured during the deployment to the **staging** slot:
```
>>> App Version <<< : 2 [Released On: 12/14/2020] - (76 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (80 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (77 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (89 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (96 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (81 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (450 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (102 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (356 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (85 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (107 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (69 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (85 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (77 ms)
ERROR: 503 - The service is unavailable. - (90 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (6043 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (104 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (74 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (92 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (106 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (73 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (70 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (98 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (97 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (100 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (89 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (91 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (85 ms)
>>> App Version <<< : 2 [Released On: 12/14/2020] - (89 ms)
```
How can we avoid failed deployment to **staging** slot bringing the **production** instance down (See the production instance ping results above with 503 error)?
Any help will be appreciated.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.