Azure / Azure/azure-functions-host
[Flex Consumption][Python] Worker shutdown during DrainMode after successful function execution is treated as an error
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
We have a Python Azure Function App running on Flex Consumption. A timer-triggered function runs once every minute and usually performs no work.
Occasionally, the function host enters `DrainMode` after a successful timer invocation/run. During the subsequent host shutdown, the Python worker exits with code 143 and this is reported as an Error-level `WorkerProcessExitException`, triggering an alert email.
#### Investigative information
- Timestamp: 2026-09-08T11:04:00.0019081Z
- Functions runtime: ~4
- Functions host version: 4.1053.200.26403
- Worker assembly: Microsoft.Azure.WebJobs.Script.Grpc, Version=4.1053.0.0
- Extension bundle configuration: `[4.*, 5.0.0)`
- Invocation Id: 06dbbe9e-d06d-4cff-a09a-5556d1e8fefe
- Region: West Europe
- Hosting plan: Azure Functions Flex Consumption
- OS: Linux
- Language: Python 3.12 (v2 programming model)
- Always-ready instances: not configured
- maximum scale out 1
We can provide the function app name and function name privately if necessary.
```text
python exited with code 143 (0x8F)
Language Worker Process exited. Pid=64.
```
Error output:
```text
Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException: at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.ThrowIfExitError
(Microsoft.Azure.WebJobs.Script.Grpc, Version=4.1053.0.0,
Culture=neutral, PublicKeyToken=null:
/_/src/WebJobs.Script.Grpc/ProcessManagement/WorkerProcess.cs:276) at Microsoft.Azure.WebJobs.Script.Workers.WorkerProcess.OnProcessExited
(Microsoft.Azure.WebJobs.Script.Grpc, Version=4.1053.0.0,
Culture=neutral, PublicKeyToken=null:
/_/src/WebJobs.Script.Grpc/ProcessManagement/WorkerProcess.cs:231)
```
The successful invocation and the later worker shutdown are separate events. In the example below, the invocation completed successfully, and the host entered DrainMode approximately 40 seconds later:
|timestamp|message|severityLevel|itemType|customDimension|
|:-------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------|----:|:----------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2026-09-08T11:04:00.0019081Z | Executing 'Functions.myfunc' (Reason='Timer fired at 2026-09-08T11:04:00.0015964+00:00', Id=06dbbe9e-d06d-4cff-a09a-5556d1e8fefe) | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Function.myfunc","prop__invocationId":"06dbbe9e-d06d-4cff-a09a-5556d1e8fefe","InvocationId":"06dbbe9e-d06d-4cff-a09a-5556d1e8fefe","prop__functionName":"Functions.myfunc","EventId":"1","EventName":"FunctionStarted","prop__reason":"Timer fired at 2026-09-08T11:04:00.0015964+00:00"} |
| 2026-09-08T11:04:00.0020912Z | Trigger Details: ScheduleStatus: {"Last":"2026-09-08T11:03:00.0002346+00:00","Next":"2026-09-08T11:04:00+00:00","LastUpdated":"2026-09-08T11:03:00.0002346+00:00"} | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Function.myfunc","InvocationId":"06dbbe9e-d06d-4cff-a09a-5556d1e8fefe","prop__ScheduleStatus":"{\"Last\":\"2026-09-08T11:03:00.0002346+00:00\",\"Next\":\"2026-09-08T11:04:00+00:00\",\"LastUpdated\":\"2026-09-08T11:03:00.0002346+00:00\"}"} |
| 2026-09-08T11:04:00.4233598Z | Executed 'Functions.myfunc' (Succeeded, Id=06dbbe9e-d06d-4cff-a09a-5556d1e8fefe, Duration=421ms) | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Function.myfunc","prop__invocationId":"06dbbe9e-d06d-4cff-a09a-5556d1e8fefe","InvocationId":"06dbbe9e-d06d-4cff-a09a-5556d1e8fefe","prop__functionName":"Functions.myfunc","EventId":"2","EventName":"FunctionCompleted","prop__status":"Succeeded","prop__executionDuration":"421"} |
| 2026-09-08T11:04:40.0390727Z | DrainMode mode enabled | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Microsoft.Azure.WebJobs.Host.DrainModeManager"} |
| 2026-09-08T11:04:40.0392468Z | Calling StopAsync on the registered listeners | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Microsoft.Azure.WebJobs.Host.DrainModeManager"} |
| 2026-09-08T11:04:40.0398882Z | Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'myfunc' | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Host.Startup","prop__functionName":"myfunc","prop__listenerName":"Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener"} |
| 2026-09-08T11:04:40.0642991Z | Stopped the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'myfunc' | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Host.Startup","prop__functionName":"myfunc","prop__listenerName":"Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener"} |
| 2026-09-08T11:04:40.066106Z | Call to StopAsync complete, registered listeners are now stopped | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Microsoft.Azure.WebJobs.Host.DrainModeManager"} |
| 2026-09-08T11:04:40.2945768Z | | 3 | exception | {"ProcessId":"44","HostInstanceId":"af740deb-...","prop__{OriginalFormat}":"Language Worker Process exited. Pid=64.","FormattedMessage":"Language Worker Process exited. Pid=64.","LogLevel":"Error","Category":"Worker.rpcWorkerProcess.python.b2ab90d5-dda2-4b9e-afe9-3b5da860a678"} |
| 2026-09-08T11:04:40.2946516Z | Language Worker Process exited. Pid=64. | 3 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Error","Category":"Worker.rpcWorkerProcess.python.b2ab90d5-dda2-4b9e-afe9-3b5da860a678"} |
| 2026-09-08T11:04:40.295268Z | Stopping JobHost | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Microsoft.Azure.WebJobs.Hosting.JobHostService"} |
| 2026-09-08T11:04:40.2954379Z | Job host stopped | 1 | trace | {"ProcessId":"44","HostInstanceId":"af740deb-...","LogLevel":"Information","Category":"Host.Startup"} |
| 2026-09-08T11:04:52.6341332Z | Initializing Warmup Extension. | 1 | trace | {"ProcessId":"45","HostInstanceId":"848fc292-ce4e-4c38-b6e6-759b37308148","LogLevel":"Information","Category":"Host.Triggers.Warmup"} |
Timer trigger:
```python
@myfunc_bp.timer_trigger(
arg_name="timer",
schedule="0 * * * * *",
run_on_startup=False,
use_monitor=True,
)
```
## Known workarounds
None known. Limiting maximum app scale-out to one instance did not prevent the issue.
## Expected behavior
The worker shutdown should not be reported as a function error or trigger a failure alert when the function invocation itself completed successfully.
Contributor guide
Research direction
Start in src/WebJobs.Script.Grpc/ProcessManagement/WorkerProcess.cs, especially the exit handling shown at lines 231 and 276, and trace how worker exits are classified while the host is in DrainMode. Verify the behavior against the reported successful invocation followed by shutdown; done means an expected shutdown no longer produces an Error-level worker exception or failure alert.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100