Azure / Azure/azure-functions-host

Did not find initialized langauge worker race condition

Open
#10,254 0 comments 0 reactions 0 assignees View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

> There's a rare edge case where the Azure Functions Host restarts the crashed Node.js process and gets into an invalid state. The Node.js worker does restart, but the host doesn't know about the new worker and fails executions with "Did not find any initialized language workers" until the host itself gets restarted or recycled.

If you look at the following query you see that the host tries to restart the crashed node process and then fails with the following error:

Execute: [[Web](https://dataexplorer.azure.com/clusters/https%3a%2f%2fwawseas.kusto.windows.net/databases/wawsprod?query=H4sIAAAAAAAEAIWRTWsCMRCG74L%2fYfBShVV3df3A4qGgpUIrUqVCLxKTt2va3WRJYq2lP77Z1VKhh96SyTMzz0zabZrAMZlCUKoTSy%2faEEQC4syCDjsY0E5bR0LDqitHBlwnSn7Cn6xjxvnMgzZvMNVKCkceHZNgDk5mqHfCTtwMe81osIr6o7gz6nVbcRiGz43rE45%2f8H5J3u4Vd1Ire%2b8dq5Wvs9jCy0iLlU9dOpbltIU7AIrq3qLV8sUbv%2fCjTjFTXllxEI3HVMtFtMxYmk4gJPcKYo3tuhylYDez%2bSbqDnrD2kVDKYrMVKuk3o37w7J8bvQruGsaaCNg%2flgFtNxnGTPHgGZKwUw%2fOPJimgdYyxIE5y%2bwHtR7w33gzm%2f8CcZ6KKDpO5Sbs8zHf%2fZwupUPCyOL2mWni%2fAsvxHC%2f1C5rZPX9vh3Yczyb0oRv9kDAgAA)] [[Desktop](https://wawseas.kusto.windows.net/wawsprod?query=H4sIAAAAAAAEAIWRTWsCMRCG74L%2fYfBShVV3df3A4qGgpUIrUqVCLxKTt2va3WRJYq2lP77Z1VKhh96SyTMzz0zabZrAMZlCUKoTSy%2faEEQC4syCDjsY0E5bR0LDqitHBlwnSn7Cn6xjxvnMgzZvMNVKCkceHZNgDk5mqHfCTtwMe81osIr6o7gz6nVbcRiGz43rE45%2f8H5J3u4Vd1Ire%2b8dq5Wvs9jCy0iLlU9dOpbltIU7AIrq3qLV8sUbv%2fCjTjFTXllxEI3HVMtFtMxYmk4gJPcKYo3tuhylYDez%2bSbqDnrD2kVDKYrMVKuk3o37w7J8bvQruGsaaCNg%2flgFtNxnGTPHgGZKwUw%2fOPJimgdYyxIE5y%2bwHtR7w33gzm%2f8CcZ6KKDpO5Sbs8zHf%2fZwupUPCyOL2mWni%2fAsvxHC%2f1C5rZPX9vh3Yczyb0oRv9kDAgAA&web=0)] https://wawseas.kusto.windows.net/wawsprod
// Detailed logs for edge case where host doesn't recognize restarted worker
let st = datetime(2024-05-17T16:42:53.4000Z);
let et = datetime(2024-05-17T16:42:56Z);
FunctionsLogs
| where PreciseTimeStamp between (st .. et)
| where RoleInstance == "pd1SmallDedicatedWebWorkerRole_IN_13758"
| where Pid == long(3468)
| project-reorder PreciseTimeStamp, Summary, InnerExceptionMessage, Details, Source, HostVersion, EventName, FunctionName, EventPrimaryStampName, EventIpAddress
| order by PreciseTimeStamp asc

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Azure Functions Host path that restarts a crashed Node.js worker, using the linked FunctionsLogs query and its timestamps as the reproduction evidence. Done means the host recognizes the restarted worker and executions no longer fail with "Did not find any initialized language workers" without requiring a host restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, node.js
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.