Azure / Azure/azure-functions-host
Language workers throwing exceptions during cold start causing extra JIT time
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
Exception is:
ExceptionType: System.InvalidOperationException
ExceptionMessage: Number of initialized language workers exceeded:1 exceeded maxProcessCount: 0
This is being seen on both node and java perfview profiles from SLA sites.
code throwing exception is here: https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs#L319
Callstack:
Name
--
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17].MoveNext()
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17].MoveNext(class System.Threading.Thread)
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17].ExecutionContextCallback(class System.Object)
+ microsoft.azure.webjobs.script!Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17.MoveNext()
+ system.private.corelib!AsyncTaskMethodBuilder.SetResult
+ system.private.corelib!System.Threading.Tasks.Task`1[System.Threading.Tasks.VoidTaskResult].TrySetResult(System.Threading.Tasks.VoidTaskResult)
+ system.private.corelib!Task.FinishContinuations
+ system.private.corelib!Task.RunContinuations
+ system.private.corelib!System.Runtime.CompilerServices.AsyncMethodBuilderCore+ContinuationWrapper.Invoke()
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22].MoveNext()
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22].MoveNext(class System.Threading.Thread)
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22].ExecutionContextCallback(class System.Object)
+ microsoft.azure.webjobs.script.webhost!Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ microsoft.azure.webjobs.script.webhost!Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__51.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ **microsoft.azure.webjobs.script.webhost!Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__62.MoveNext()**
+ microsoft.extensions.hosting.il!Host.StopAsync
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ microsoft.extensions.hosting!Microsoft.Extensions.Hosting.Internal.Host+d__10.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ **microsoft.azure.webjobs.script!Microsoft.Azure.WebJobs.Script.Workers.FunctionInvocationDispatcherShutdownManager+d__3.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ system.private.corelib!System.Runtime.CompilerServices.TaskAwaiter`1[System.__Canon].GetResult()
+ system.private.corelib!TaskAwaiter.HandleNonSuccessAndDebuggerNotification
+ system.private.corelib!ExceptionDispatchInfo.Throw
Contributor guide
Research direction
Start at src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs line 319 and trace the SpecializeAsync call path shown in the call stack. Compare the cold-start behavior for the Node and Java worker profiles described in the issue. Done means cold start no longer throws the maxProcessCount exception or incurs the reported extra JIT time; no test file is named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100