Azure / Azure/azure-functions-powershell-worker

PowerShell language worker support for cancellation token that is emitted by the function host.

Open
#949 0 comments 0 reactions 0 assignees View on GitHub
P2
Dominant language
C#
Stars
215
Forks
61
Avg merge
21h 4m
Merged PRs (30d)
6

Description

There is a scenario where one was getting "Task Cancelled exception " in PowerShell functions.

Entire exception trace:

**System.Threading.Tasks.TaskCanceledException : A task was canceled.**
   at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 96
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs : 20
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 52
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 581
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 527
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 306
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 352
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 108

I could see that at the time of failure, there were no worker processes ( powershell processes ) getting assigned to the invocations.
However, when we see for successful executions, there are processes attached correctly. Invocation Response received correctly

Since this is a queue trigger message, so the messages are getting retried and no loss of functionality... 

To remediate this, there was a suggestion to handle the cancellation token as mentioned here ( this is for .Net ):
[Improve error handling for invocation request cancellations · Issue #8986 · Azure/azure-functions-host (github.com)](https://github.com/Azure/azure-functions-host/issues/8986)

However, the powershell language worker hasn't been ready to support cancellation token that is emitted by the function host.
as per https://github.com/Azure/azure-functions-host/wiki/Out-of-proc-Support-for-Cancellation-Tokens

Requesting you to please create a roadmap for this feature.

Looping in @AnatoliB , Thank you for the information.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the WorkerFunctionInvoker.cs call site cited in the exception trace and read the linked out-of-proc cancellation-token support guidance. Determine how the PowerShell worker should receive the function host's cancellation token; done means the worker supports that token and avoids the reported TaskCanceledException scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, powershell
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.