Azure / Azure/azure-functions-host
Recurring instance cycling causing messages to spend long time in sb topic subscription before being handled
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
Hi!
I've been investigating sporadic long processing times in one of our functions only to realise that vast majority of time is actually spent elsewhere in a recurring pattern. The function in this case uses a service bus trigger (topic subscription) and it hums a long and handles a message every few seconds and spends the majority of its time scaled to one instance on a consumption plan. Whenever Azure decides to cycle instances it would appear that the current instance is (drain) stopped and depending on luck a new instance is started immediately and a normal cold start penalty of a few seconds is observed (this is fine and NOT the issue) or in the case of bad luck there may be a delay of up to 60-70 seconds before an instance is started.
The screenshot shows the maximum time spent in queue in seconds per instance. Time spent in queue is calculated by subtracting EnqueuedTimeUtc from the timestamp of the trace log entry.

I may be interpreting things the wrong way and in that case please let me know. If not then it would be ideal if the scaling mechanism could immediately start a new instance if the last instance is being (drain) stopped due to the recurring instance shuffle. The [documentation](https://docs.microsoft.com/en-us/azure/azure-functions/event-driven-scaling#understanding-scaling-behaviors) states that `for non-HTTP triggers, new instances are allocated, at most, once every 30 seconds`. As seen in the screenshot around 4:30 am a new instance is started immediately and processing of messages continues with little time spent in queue. The other instances are not that lucky. I'm guessing there is another mechanism in play when there are no instances allocated?
The invocation (15fc6812-e6d8-4f6d-8ecd-6f60c9b626cb) takes 8.3 seconds due to cold start (which is fine and not the issue). Time spent in queue before this invocation is 69 seconds.
#### Investigative information
Please provide the following:
- Timestamp: 2022-07-07T00:00:00Z - 2022-07-07T12:00:00Z
- Function App version: 4
- Function App name:
- Function name(s) (as appropriate):
- Invocation ID: 15fc6812-e6d8-4f6d-8ecd-6f60c9b626cb
- Region: Europe West
Contributor guide
Assessment
This issue has not been assessed yet.