Azure / Azure/azure-functions-python-worker
Nondeterministc deploy bug with 5+ different non-descript failures on two different OS's, worked before, support stumped
- Dominant language
- Python
- Stars
- 357
- Forks
- 116
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
Unsure how to title this. It's been a week of debugging and isolating with no results.
Is this issue in the correct repo? Also unsure. Help me out here.
I'll walk you through my journey.
I have a very large code base, it runs flawlessly locally. I've deployed it just fine for a long time until recently.
I made a change, including `google-cloud-texttospeech` in `requirements.txt`.
It stopped deploying after this (maybe relevant, maybe not).
Removing this change, to the **exact same code base** as before, still fails to deploy.
Some errors I get at random, and I've tried incredibly hard to isolate.
Both development environments can deploy other things successfully, and push to a new app fine as well.
I have not made any local environment changes at all when it began to fail.
**Details:**
Azure Functions runtime version 4.24.4.4, Linux premium plan with Elastic Premium EP3.
I deploy directly from visual studio code azure function extension.
My python code follows the folder structure of the v1 coding model (no decorators, lots of folders), however I am on v2 (host.json etc.), this has always worked, and runs locally of course.
Deploying to python 3.9.7.
**First development environment:**
Manjaro linux, vscode 1.81.1, azure extension 1.12.3
Errors I've encountered seemingly at random, when trying to deploy:
```
2:50:41 PM debugApp123: **Deployment successful**. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
2:51:07 PM debugApp123: Syncing triggers...
2:51:12 PM debugApp123: Querying triggers...
2:51:18 PM debugApp123: **No HTTP triggers found.**
```
```
3:34:11 PM debugApp123: Deployment Failed. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
3:34:25 PM debugApp123: Deployment failed.
```
```
4:00:49 PM: Error: The operation was aborted.
```
```
10:52:10 AM ae-API-compute: Deployment successful. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
10:52:42 AM ae-API-compute: Syncing triggers...
10:52:49 AM ae-API-compute: Querying triggers...
10:52:52 AM ae-API-compute: WARNING: Some http trigger urls cannot be displayed in the output window because they require an authentication token. Instead, you may copy them from the Azure Functions explorer.
```
Except it was not successful and functions are empty / don't run (the zip file in webjobstorage contains them thought).
```
11:55:04 AM debugApp123: Syncing triggers...
11:55:45 AM debugApp123: Syncing triggers (Attempt 2/6)...
11:55:56 AM debugApp123: Syncing triggers (Attempt 3/6)...
11:56:17 AM debugApp123: Syncing triggers (Attempt 4/6)...
11:56:59 AM debugApp123: Syncing triggers (Attempt 5/6)...
11:58:20 AM debugApp123: Syncing triggers (Attempt 6/6)...
11:59:12 AM: Error: Encountered an error (ServiceUnavailable) from host runtime.
```
And a very exciting shiny rare one:
```
Offset to Central Directory cannot be held in an Int64.
```
**Second development environment:**
Windows, visual studio code, azure extension v1.12.4.
I've also used this environment before, and it has worked, same as the above one.
Errors:
One successful deploy, and randomly most of the above (with no changes), as well as a new one:
```
10:35:34 AM ae-API-compute: Starting deployment...
10:35:34 AM ae-API-compute: Creating zip package...
10:45:11 AM: Error: socket hang up
```
**Some other random things I've tried:**
- different development environments with different OS's, cloning the repo fresh
- rolling back azure function extension in visual studio code to various versions
- protobuf==3.20.*
- setting AzureWebJobsFeatureFlags:EnableWorkerIndexing
- setting SCM_DO_BUILD_DURING_DEPLOYMENT:1
- removing type hints
- restarting
**What now?**
The errors are not helpful. Logs are missing, I can drill down into events, insights, many different logs, they are all over the place and they are all useless or empty. I've mentioned that I've had multiple 2hr+ calls with the technical support team of various ever-increasing escalations, and they are just as stumped as me (and I am grateful for their efforts).
**Any thoughts?**
What do I try next? Any information I can provide?
**See also** (maybe relevant, I don't know at this point):
https://github.com/microsoft/vscode-azurefunctions/issues/3805
https://github.com/microsoft/vscode-azurefunctions/issues/2529
https://github.com/microsoft/Oryx/issues/1774
https://stackoverflow.com/questions/76478668/adding-python-module-google-cloud-storage-is-causing-a-working-azure-function-ap
https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly
https://github.com/projectkudu/kudu/issues/3348
https://github.com/microsoft/azure-pipelines-tasks/issues/14201
https://github.com/microsoft/vscode-azurefunctions/issues/2529
https://github.com/microsoft/Oryx/issues/1774
Contributor guide
Assessment
This issue has not been assessed yet.