Azure / Azure/azure-functions-host

Is the a way to extend the shutdown timeout in Azure function so that the drain has enough time to finish?

Open
#9,822 1 comment 2 reactions 0 assignees View on GitHub
needs-discussion Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

We use the Function App to process time-consuming operations including heavy database queries & sending large files to some 3rd party. We host them using `dotnet-isolated` runtime in Azure Kubernetes Services (AKS) using [the official Docker image ](https://hub.docker.com/_/microsoft-azure-functions-dotnet-isolated). Sometimes execution of the function takes 1-2 minutes and it's quite undesired to stop it in the middle of processing. It looks like the proper approach to maximize the chance that the function finishes the execution is to drain the function host on exiting.

After some research, I found that there is the `FUNCTIONS_ENABLE_DRAIN_ON_APP_STOPPING` environment variable enabling this feature but then I encountered another issues:
- The Docker container doesn't pass the termination signal to the dotnet host function because it uses the bash wrapper. I've tried to correct it in the https://github.com/Azure/azure-functions-docker/pull/1023
- I didn't find a way to override [the ProcessShutdownTimeout](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcessCountOptions.cs#L58). Is there way to do this?
- The [drainModeManager.EnableDrainModeAsync(CancellationToken.None)](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L872) didn't finish before the process exists. I've tried to fix it by https://github.com/Azure/azure-functions-host/pull/9780 but then I checked it and it also didn't help. After that I've tried calling this synchronously:
```
drainModeManager.EnableDrainModeAsync(CancellationToken.None).GetAwaiter().GetResult();
```
and now it waits till the function really finishes

Contributor guide

Open the contributing guide

Research direction

Start with WorkerProcessCountOptions.cs and the ProcessShutdownTimeout setting, then trace the shutdown path from WebJobsScriptHostService.cs and drainModeManager.EnableDrainModeAsync. Review the Docker image's bash wrapper and the referenced PRs 1023 and 9780. Done means the host receives termination, drains reliably, and the shutdown timeout can be overridden for this deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, bash, csharp, docker, kubernetes
Domain
backend, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.