Azure / Azure/azure-functions-host
Out-of-proc worker restart should signal host to kill all function invocation timers.
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
**Current Behavior**
In an out-of-proc language (ex. Python) when a function invocation reaches the `functionTimeout` limit, the worker process is restarted. Any in-flight invocations running on that process are also terminated. However, the out of proc worker may not signal the host to kill execution timers for all invocations that were terminated. We observe:
1. Timer Function A starts at 00:00:00.
2. Timer Function B starts at 00:02:30.
3. The Functions host throws a FunctionTimeoutException for Function A at 00:05:00.
4. The language worker process terminates all of its function invocations and restarts.
5. The Functions host throws a FunctionTimeoutException for Function B at 00:07:30.
6. The language worker process terminates all of its function invocations and restarts.
This can lead to additional billing based on execution time.
**Desired Behavior**
When a function invocation reaches the `functionTimeout` limit, the host should signal the out-of-proc language worker to terminate all function invocations AND send a response with an error back to the host so it can kill all related function invocation timers. We should observe:
1. Timer Function A starts at 00:00:00.
2. Timer Function B starts at 00:02:30.
3. The Functions host throws a FunctionTimeoutException for Function A at 00:05:00.
4. The language worker process terminates all of its function invocations and restarts.
**Additional Context**
Reproduced here:
- Timestamp: 2022-08-16 11:48:20 to 2022-08-16 11:54:30
- Function App version: 4.9.1.1
- Invocation ID: 4287c381-3dab-4412-8eb8-db3731b856be and b99d7a5d-e145-404a-becb-c098f8f0e8c0
- Region: East Asia
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing out-of-proc worker timeout and restart handling alongside the host's function invocation timers; done means a worker restart terminates all affected invocations and the host clears their timers without later timeout exceptions or extra execution time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100