Azure / Azure/azure-functions-host

QueueTrigger fails with a TaskCanceledException after being idle

Open
#1,499 0 comments 1 reaction 1 assignee Claimed by @mamaso View on GitHub
needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

We have a Queue function that takes an incoming ID via a storage table queue and connects to Azure SQL server and an external mail system to relay emails. If the queue has been idle for a couple hours (still trying to pin down the exact amount of time), the next invocation immediately fails with a TaskCanceledException. Because of how the dequeue works, this message is now unavailable for the next 10 minutes until the storage table queue enables it to be resent. After that time, the message is processed and everything is happy. Our average execution time is around 800ms, but we've seen it get up to about 4 seconds before, much shorter than when a cancellation would occur.

#### Repro steps
1. Have a queue trigger function that takes a CancellationToken and passes it further in the framework or call `ThrowIfCancellationRequested()`.
2. Add an item in the queue to warm it up.
3. Wait a long time (around half a day or so).
4. Insert another queue item and watch it except.

#### Expected behavior
If a function recycle is necessary, the function should return the item to the queue before tearing down, so the storage queue timeout is not observed. Even better, just process the item without recycling.

#### Actual behavior
After a long idle period, the function throws the exception and the original queue message is unavailable until the default queue timeout expires (default: 10 minutes).

#### Known workarounds
Recycle the application before queuing the first message.

#### Related information
This is a C# .csx file that runs as a script. The configuration is below, as is the full stack trace as captured by Application Insights. The function runs on a consumption plan located in West US 2.

`function.json`:
```json
{
"disabled": false,
"bindings": [
{
"name": "myQueueItem",
"type": "queueTrigger",
"direction": "in",
"queueName": "email-submission",
"connection": "tasks_STORAGE"
}
]
}
```

Exception:
```text
System.Threading.Tasks.TaskCanceledException: Exception while executing function: Functions.SendEmailQueueMuncher
Exception has been thrown by the target of an invocation. One or more errors occurred. A task was canceled.
Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__14.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__12.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception System.Reflection.TargetInvocationException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
at System.RuntimeMethodHandle.InvokeMethod (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Reflection.RuntimeMethodInfo.Invoke (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Reflection.RuntimePropertyInfo.GetValue (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Reflection.RuntimePropertyInfo.GetValue (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker.GetTaskResult (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker+<>c.b__23_0 (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task.Execute (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker+d__23.MoveNext (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+d__33.MoveNext (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1+d__8.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__23.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__22.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__14.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Inner exception System.AggregateException handled at System.RuntimeMethodHandle.InvokeMethod:
at System.Threading.Tasks.Task.ThrowIfExceptional (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task`1.GetResultCore (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.Tasks.Task`1.get_Result (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
Inner exception System.Threading.Tasks.TaskCanceledException handled at System.Threading.Tasks.Task.ThrowIfExceptional:
```

There seems to be a related exception, however it's not always present. It may be related though.
```text
An unhandled exception has occurred. Host is shutting down.
System.ObjectDisposedException:
at System.Threading.CancellationTokenSource.ThrowObjectDisposedException (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.CancellationTokenSource.InternalRegister (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.CancellationToken.Register (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Threading.CancellationToken.Register (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer+d__11.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener+d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.