Azure / Azure/azure-functions-host
Potential TimerTrigger deadlock on Elastic Premium plans
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
_Just creating this issue for tracking, and to follow-up on an internal incident_
We recently investigated an incident where a customer's TimerTrigger Function stopped executing until the application restarted. Before the restart, the last invocation of the TimerTrigger resulted in a time-out, which led the language worker to restart **but the Host remained online**. It appears that since the Host remained online, the TimerTrigger lease for the failed Function never got released; this seems to be the reason why the TimerTrigger stopped executing until the restart took place. When the restart took place, our logs indicate that the TimerTrigger lease finally got released and so, after the restart completed, the Function continued executing as expected.
The plan was Elastic Premium, the language was PowerShell.
After discussing with @fabiocav, we believe a simple reproducer can be written as follows:
(1) Create a TimerTrigger Function "A" to run every minute
(2) Make "A" pause for "5 minutes"
(3) Force the TimerTrigger timeout to be 2 minutes
This specific configuration should replicate, on a smaller total time window, the sequence of events we observed on the incident.
Then, observe what code path is taken and ensure the lease is released.
Contributor guide
Research direction
Start by reproducing the reported sequence with a PowerShell TimerTrigger that runs every minute, pauses for five minutes, and has a two-minute timeout on an Elastic Premium plan. Trace the TimerTrigger timeout and lease-handling path, then confirm that the lease is released while the Host remains online and the function can execute again without an application restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, powershell
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100