Azure / Azure/azure-functions-core-tools
Unexpected Python Timeout Behaviour
- Dominant language
- C#
- Stars
- 1.5k
- Forks
- 498
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 14
Description
Hi, I am trying to understand how exactly the timeouts are handled in python based azure functions. We are running the following scenario:
- running "func start" on a virtual machine with process count == 8 and thread count == 12 and a timeout of 10 minutes
- running a queue based function running with python 3.8
- messages are processed concurrently and using a shared threadlock for thread unsafe objects (all locks are used in a with context)
The issue that we are seeing is that if one thread of a process runs into a timeout that thread is killed, as expected, but all other threads of that process stay open and running until their respective timeout is also reached. They do not finish processing their messages and seem to be deadlocked or sleeping.
My question is the following, what is actually going on here? Why does the azure function worker not kill the process directly and end all threads? Why does the thread that is killed not release the lock properly?
Contributor guide
Research direction
Start by reproducing the queue-function scenario with `func start`, Python 3.8, eight processes, twelve threads, and the ten-minute timeout described in the issue. Observe the other threads and shared locks when one thread times out; the work is done when the timeout, process, thread, and lock behavior is explained from the reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100