Azure / Azure/azure-functions-python-worker
[Bug] function execution hanging
- Dominant language
- Python
- Stars
- 357
- Forks
- 116
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
### Expected Behavior
Hi
Over the last few months I've noted an issue where functions (both timer and service bus triggered) will occasionally hang mid-execution. They silently stop, do no more work and produce no more logs until they are killed by the host after hitting the global timeout.
### Actual Behavior
I've noticed two things happening in conjunction with this error, the error below: (I have set `PYTHONASYNCIODEBUG=1`)
```
[2025-11-17T10:19:22.478Z] Task was destroyed but it is pending!
[2025-11-17T10:19:22.478Z] source_traceback: Object created at (most recent call last):
[2025-11-17T10:19:22.478Z] File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/worker.py", line 71, in
[2025-11-17T10:19:22.478Z] main.main()
[2025-11-17T10:19:22.478Z] File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/main.py", line 61, in main
[2025-11-17T10:19:22.478Z] return asyncio.run(start_async(
[2025-11-17T10:19:22.478Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 190, in run
[2025-11-17T10:19:22.478Z] return runner.run(main)
[2025-11-17T10:19:22.478Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run
[2025-11-17T10:19:22.478Z] return self._loop.run_until_complete(task)
[2025-11-17T10:19:22.478Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
[2025-11-17T10:19:22.478Z] self.run_forever()
[2025-11-17T10:19:22.479Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
[2025-11-17T10:19:22.479Z] self._run_once()
[2025-11-17T10:19:22.479Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1928, in _run_once
[2025-11-17T10:19:22.479Z] handle._run()
[2025-11-17T10:19:22.479Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/events.py", line 84, in _run
[2025-11-17T10:19:22.479Z] self._context.run(self._callback, *self._args)
[2025-11-17T10:19:22.479Z] File "/home/alexander/.local/share/uv/python/cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 443, in create_task
[2025-11-17T10:19:22.479Z] task = self._task_factory(self, coro)
[2025-11-17T10:19:22.479Z] File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py", line 184, in
[2025-11-17T10:19:22.479Z] lambda loop, coro, context=None: ContextEnabledTask(
[2025-11-17T10:19:22.479Z] File "/usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py", line 1096, in __init__
[2025-11-17T10:19:22.479Z] super().__init__(coro, loop=loop, context=context)
[2025-11-17T10:19:22.479Z] task: wait_for= created at /usr/lib/azure-functions-core-tools-4/workers/python/3.11/LINUX/X64/azure_functions_worker/dispatcher.py:1096>
```
Also, within my function, a `GeneratorExit` is thrown.
Because the `GeneratorExit` is not an `Exception`, the error is not handled by the dispatcher [here](https://github.com/Azure/azure-functions-python-worker/blob/ea12fd28faaae3e68c558ec3dcba52ecbf7397f3/workers/azure_functions_worker/dispatcher.py#L734) so the host never knows that the worker has crashed.
### Steps to Reproduce
I believe this is related to timing of GC sweeps so I have not been able to deterministically reproduce the issue. However I've managed to reliably reproduce it locally by setting up one function that runs every minute and spins up nonsense tasks:
```
async def pretend_work():
log.info("running task")
await asyncio.sleep(random.randint(0, 30) + 30)
log.info("stopping task")
async def main(mytimer: TimerRequest) -> None:
tasks = (pretend_work() for _ in range(0, 500))
await asyncio.gather(*tasks)
```
And a second function that connects to a service bus server and sends a message. After a few minutes of running this, the second function will stop executing, raise the asyncio error above and do nothing more.
Setting the gc to be more aggressive with `gc.set_threshold(1, 1, 1)` seems to increase the likelihood of triggering the error.
### Relevant code being tried
```shell
```
### Relevant log output
```shell
```
### requirements.txt file
```shell
Using Core tools 4.4.1 and v1 programming model. Python 3.11.
```
### Where are you facing this problem?
Production Environment (explain below)
### Function app name
_No response_
### Additional Information
I believe the issue is caused by the asyncio tasks created by the worker getting GCd because there are no references remaining to them.
https://github.com/Azure/azure-functions-python-worker/blob/ea12fd28faaae3e68c558ec3dcba52ecbf7397f3/workers/azure_functions_worker/dispatcher.py#L1060
This is mentioned in the [asyncio docs](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task):
> Important Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done. For reliable “fire-and-forget” background tasks, gather them in a collection:
Contributor guide
Assessment
This issue has not been assessed yet.