Azure / Azure/azure-functions-host

Language workers throwing exceptions during cold start causing extra JIT time

Open
#7,342 3 comments 0 reactions 0 assignees View on GitHub
bug feature:language-extensibility perf: cold-start
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

Exception is:
ExceptionType: System.InvalidOperationException
ExceptionMessage: Number of initialized language workers exceeded:1 exceeded maxProcessCount: 0

This is being seen on both node and java perfview profiles from SLA sites.

code throwing exception is here: https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs#L319

Callstack:

Name
--
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17].MoveNext()
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17].MoveNext(class System.Threading.Thread)
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17].ExecutionContextCallback(class System.Object)
+ microsoft.azure.webjobs.script!Microsoft.Azure.WebJobs.Script.Workers.Rpc.WebHostRpcWorkerChannelManager+d__17.MoveNext()
+ system.private.corelib!AsyncTaskMethodBuilder.SetResult
+ system.private.corelib!System.Threading.Tasks.Task`1[System.Threading.Tasks.VoidTaskResult].TrySetResult(System.Threading.Tasks.VoidTaskResult)
+ system.private.corelib!Task.FinishContinuations
+ system.private.corelib!Task.RunContinuations
+ system.private.corelib!System.Runtime.CompilerServices.AsyncMethodBuilderCore+ContinuationWrapper.Invoke()
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22].MoveNext()
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22].MoveNext(class System.Threading.Thread)
+ system.private.corelib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22].ExecutionContextCallback(class System.Object)
+ microsoft.azure.webjobs.script.webhost!Microsoft.Azure.WebJobs.Script.WebHost.StandbyManager+d__22.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ microsoft.azure.webjobs.script.webhost!Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__51.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ **microsoft.azure.webjobs.script.webhost!Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__62.MoveNext()**
+ microsoft.extensions.hosting.il!Host.StopAsync
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ microsoft.extensions.hosting!Microsoft.Extensions.Hosting.Internal.Host+d__10.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ **microsoft.azure.webjobs.script!Microsoft.Azure.WebJobs.Script.Workers.FunctionInvocationDispatcherShutdownManager+d__3.MoveNext()
+ system.private.corelib.il!AsyncTaskMethodBuilder.Start
+ system.private.corelib.il!AsyncMethodBuilderCore.Start
+ system.private.corelib!System.Runtime.CompilerServices.TaskAwaiter`1[System.__Canon].GetResult()
+ system.private.corelib!TaskAwaiter.HandleNonSuccessAndDebuggerNotification
+ system.private.corelib!ExceptionDispatchInfo.Throw

Contributor guide

Open the contributing guide

Research direction

Start at src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs line 319 and trace the SpecializeAsync call path shown in the call stack. Compare the cold-start behavior for the Node and Java worker profiles described in the issue. Done means cold start no longer throws the maxProcessCount exception or incurs the reported extra JIT time; no test file is named.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.