Azure / Azure/azure-functions-host

Python language worker crashes when executing functions

Open
#3,549 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

All requests to the container failed with the below exception.

West US
FunctionsLogs
| where RoleInstance contains "171AE528-636740280065455456"
| where TIMESTAMP >= datetime(2018-10-01 23:23:00.0000000)
| where TIMESTAMP <= datetime(2018-10-01 23:23:00.0000000) + 30m
| where Tenant == "fe133483000c4137a26aabfc6365308c"
| where SourceNamespace == "LWAWS"
| where Level < 4

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.scalethis ---\u003e Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerChannelException : Failed to start language worker for: python ---\u003e System.AggregateException : One or more errors occurred. (bash exited with code 137 ) (bash exited with code 137 ) (bash exited with code 137 ) ---\u003e bash exited with code 137 ---\u003e (Inner Exception #0) Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 \u003c--- ---\u003e (Inner Exception #1) Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 \u003c--- ---\u003e (Inner Exception #2) Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 \u003c--- End of inner exception
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionGenerator.cs : 225
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs : 63
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs : 63
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionGenerator.cs : 225
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs : 508
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs : 444
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs : 249 End of inner exception
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs : 293
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs : 89

Contributor guide

Open the contributing guide

Research direction

Start with the Python language-worker startup path in WorkerLanguageInvoker.cs and FunctionInvokerBase, using the supplied FunctionsLogs query and the repeated bash exit-code-137 trace as the failure signal. Trace the request through FunctionGenerator and determine what allows Python function execution to complete without the worker crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.