Azure / Azure/azure-functions-host
Intermittent TaskCanceledExceptions when F# function calls EventHubs GetRuntimeInformationAsync
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
This originated with a CRI and is pretty easy to reproduce. The set up is to have an F# HttpTrigger function that uses keyvault to store an eventhubs connectionstring in the AppSettings and calls `GetRuntimeInformationAsync()` with an `EventHubClient`. The function endpoint is then called every 5 minutes by an Azure Availability Test. Often, maybe 50 - 100 time per day, it fails with this stack trace:
```
System.Threading.Tasks.TaskCanceledException : A task was canceled.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 52
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 585
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 532
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 468
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 278
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 322
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 117
```
The Function:
```
module HttpTrigger =
[]
let eventHubConnectionString = Environment.GetEnvironmentVariable(eventHubConnectionStringKey, EnvironmentVariableTarget.Process)
let eventHubClient = lazy(EventHubClient.CreateFromConnectionString(eventHubConnectionString))
[]
let checkEventHubs ([]req: HttpRequest) (log: ILogger) =
async {
let! information = eventHubClient.Force().GetRuntimeInformationAsync() |> Async.AwaitTask
printfn "%A" (System.String.Format("{0:D}", information))
return OkObjectResult("…") :> IActionResult
} |> Async.StartAsTask
```
The EventHubs team said they had looked into the logs from their side and they didn't see any canceled requests or time outs.
Things I've tried:
- Reached out to an F# expert who gave me the proper way to structure this function, but it still throws exceptions.
- Removing KeyVault from the equation by pasting the connection string into the code, this gave the same exceptions.
- Reproduced this functionality in C#, I've yet to see an exception in the ~week I've been running it
- Made an almost identical function that calls ServiceBus `SendAsync()` instead of EventHubs. This has never thrown an exception.
- Removed the availability test and instead called the function using a postman runner every 5 minutes for multiple days. Never threw an exception.
- Switched the function to a TimerTrigger thinking it might be the availability test, this throws the same exceptions.
- As yet I'm unable to reproduce the exception locally using the TimerTrigger version.
- Made a TimerTrigger that calls the HttpTrigger (adding a step of indirection). This didn't work.
Contributor guide
Assessment
This issue has not been assessed yet.