Azure / Azure/azure-functions-host
Is the a way to extend the shutdown timeout in Azure function so that the drain has enough time to finish?
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
We use the Function App to process time-consuming operations including heavy database queries & sending large files to some 3rd party. We host them using `dotnet-isolated` runtime in Azure Kubernetes Services (AKS) using [the official Docker image ](https://hub.docker.com/_/microsoft-azure-functions-dotnet-isolated). Sometimes execution of the function takes 1-2 minutes and it's quite undesired to stop it in the middle of processing. It looks like the proper approach to maximize the chance that the function finishes the execution is to drain the function host on exiting.
After some research, I found that there is the `FUNCTIONS_ENABLE_DRAIN_ON_APP_STOPPING` environment variable enabling this feature but then I encountered another issues:
- The Docker container doesn't pass the termination signal to the dotnet host function because it uses the bash wrapper. I've tried to correct it in the https://github.com/Azure/azure-functions-docker/pull/1023
- I didn't find a way to override [the ProcessShutdownTimeout](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/ProcessManagement/WorkerProcessCountOptions.cs#L58). Is there way to do this?
- The [drainModeManager.EnableDrainModeAsync(CancellationToken.None)](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L872) didn't finish before the process exists. I've tried to fix it by https://github.com/Azure/azure-functions-host/pull/9780 but then I checked it and it also didn't help. After that I've tried calling this synchronously:
```
drainModeManager.EnableDrainModeAsync(CancellationToken.None).GetAwaiter().GetResult();
```
and now it waits till the function really finishes
Contributor guide
Research direction
Start with WorkerProcessCountOptions.cs and the ProcessShutdownTimeout setting, then trace the shutdown path from WebJobsScriptHostService.cs and drainModeManager.EnableDrainModeAsync. Review the Docker image's bash wrapper and the referenced PRs 1023 and 9780. Done means the host receives termination, drains reliably, and the shutdown timeout can be overridden for this deployment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, bash, csharp, docker, kubernetes
- Domain
- backend, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100