Azure / Azure/azure-functions-host

Out-of-proc worker restart should signal host to kill all function invocation timers.

Open
#8,720 0 comments 0 reactions 0 assignees View on GitHub
area: out-of-proc enhancement
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

**Current Behavior**
In an out-of-proc language (ex. Python) when a function invocation reaches the `functionTimeout` limit, the worker process is restarted. Any in-flight invocations running on that process are also terminated. However, the out of proc worker may not signal the host to kill execution timers for all invocations that were terminated. We observe:

1. Timer Function A starts at 00:00:00.
2. Timer Function B starts at 00:02:30.
3. The Functions host throws a FunctionTimeoutException for Function A at 00:05:00.
4. The language worker process terminates all of its function invocations and restarts.
5. The Functions host throws a FunctionTimeoutException for Function B at 00:07:30.
6. The language worker process terminates all of its function invocations and restarts.

This can lead to additional billing based on execution time.

**Desired Behavior**
When a function invocation reaches the `functionTimeout` limit, the host should signal the out-of-proc language worker to terminate all function invocations AND send a response with an error back to the host so it can kill all related function invocation timers. We should observe:

1. Timer Function A starts at 00:00:00.
2. Timer Function B starts at 00:02:30.
3. The Functions host throws a FunctionTimeoutException for Function A at 00:05:00.
4. The language worker process terminates all of its function invocations and restarts.

**Additional Context**
Reproduced here:

- Timestamp: 2022-08-16 11:48:20 to 2022-08-16 11:54:30
- Function App version: 4.9.1.1
- Invocation ID: 4287c381-3dab-4412-8eb8-db3731b856be and b99d7a5d-e145-404a-becb-c098f8f0e8c0
- Region: East Asia

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing out-of-proc worker timeout and restart handling alongside the host's function invocation timers; done means a worker restart terminates all affected invocations and the host clears their timers without later timeout exceptions or extra execution time.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.