Azure / Azure/azure-functions-host
Did not find initialized langauge worker race condition
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
> There's a rare edge case where the Azure Functions Host restarts the crashed Node.js process and gets into an invalid state. The Node.js worker does restart, but the host doesn't know about the new worker and fails executions with "Did not find any initialized language workers" until the host itself gets restarted or recycled.
If you look at the following query you see that the host tries to restart the crashed node process and then fails with the following error:
Execute: [[Web](https://dataexplorer.azure.com/clusters/https%3a%2f%2fwawseas.kusto.windows.net/databases/wawsprod?query=H4sIAAAAAAAEAIWRTWsCMRCG74L%2fYfBShVV3df3A4qGgpUIrUqVCLxKTt2va3WRJYq2lP77Z1VKhh96SyTMzz0zabZrAMZlCUKoTSy%2faEEQC4syCDjsY0E5bR0LDqitHBlwnSn7Cn6xjxvnMgzZvMNVKCkceHZNgDk5mqHfCTtwMe81osIr6o7gz6nVbcRiGz43rE45%2f8H5J3u4Vd1Ire%2b8dq5Wvs9jCy0iLlU9dOpbltIU7AIrq3qLV8sUbv%2fCjTjFTXllxEI3HVMtFtMxYmk4gJPcKYo3tuhylYDez%2bSbqDnrD2kVDKYrMVKuk3o37w7J8bvQruGsaaCNg%2flgFtNxnGTPHgGZKwUw%2fOPJimgdYyxIE5y%2bwHtR7w33gzm%2f8CcZ6KKDpO5Sbs8zHf%2fZwupUPCyOL2mWni%2fAsvxHC%2f1C5rZPX9vh3Yczyb0oRv9kDAgAA)] [[Desktop](https://wawseas.kusto.windows.net/wawsprod?query=H4sIAAAAAAAEAIWRTWsCMRCG74L%2fYfBShVV3df3A4qGgpUIrUqVCLxKTt2va3WRJYq2lP77Z1VKhh96SyTMzz0zabZrAMZlCUKoTSy%2faEEQC4syCDjsY0E5bR0LDqitHBlwnSn7Cn6xjxvnMgzZvMNVKCkceHZNgDk5mqHfCTtwMe81osIr6o7gz6nVbcRiGz43rE45%2f8H5J3u4Vd1Ire%2b8dq5Wvs9jCy0iLlU9dOpbltIU7AIrq3qLV8sUbv%2fCjTjFTXllxEI3HVMtFtMxYmk4gJPcKYo3tuhylYDez%2bSbqDnrD2kVDKYrMVKuk3o37w7J8bvQruGsaaCNg%2flgFtNxnGTPHgGZKwUw%2fOPJimgdYyxIE5y%2bwHtR7w33gzm%2f8CcZ6KKDpO5Sbs8zHf%2fZwupUPCyOL2mWni%2fAsvxHC%2f1C5rZPX9vh3Yczyb0oRv9kDAgAA&web=0)] https://wawseas.kusto.windows.net/wawsprod
// Detailed logs for edge case where host doesn't recognize restarted worker
let st = datetime(2024-05-17T16:42:53.4000Z);
let et = datetime(2024-05-17T16:42:56Z);
FunctionsLogs
| where PreciseTimeStamp between (st .. et)
| where RoleInstance == "pd1SmallDedicatedWebWorkerRole_IN_13758"
| where Pid == long(3468)
| project-reorder PreciseTimeStamp, Summary, InnerExceptionMessage, Details, Source, HostVersion, EventName, FunctionName, EventPrimaryStampName, EventIpAddress
| order by PreciseTimeStamp asc
Contributor guide
Research direction
Start by tracing the Azure Functions Host path that restarts a crashed Node.js worker, using the linked FunctionsLogs query and its timestamps as the reproduction evidence. Done means the host recognizes the restarted worker and executions no longer fail with "Did not find any initialized language workers" without requiring a host restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, node.js
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100