Azure / Azure/azure-functions-host
Intelligent concurrency for language workers
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
## Overview
The idea of the feature is to provide better concurrency for language workers depending on customers' scenarios by removing concurrency limits and automatic concurrency adjustment. Currently we have a lot of CRIs when we recommend the customers to change concurrency settings depending on their scenarios. The feature will allow them to utilize VM resources better and improve overall performance for OOP function apps without experimenting with concurrency settings.
## Goals
- **Simplify configuration**: The customers do not need to tweak OOP layer concurrency settings such as FUNCTIONS_WORKER_PROCESS_COUNT, PYTHON_THREADPOOL_THREAD_COUNT, PSWorkerInProcConcurrencyUpperBound. OOP layer can adjust the concurrency settings automatically to provide the best performance possible.
- **Removing concurrency limits**: We artificially limit the number of concurrent threads for two language workers – Python and PowerShell. Removing the limits significantly improves throughput for IO bound sync scenarios.
- **Automatically adding language workers**: Monitoring language health state and adding new workers language if needed. This will improve CPU consumption for CPU bound scenarios for Node, Python and PowerShell. Also it is only way to add concurrency for Node.
- **Load balancing invocation requests across language workers**: Currently we are using round robin to send invocation requests to language workers. The load balancing will allow sending the invocation request to a worker which has less active invocations.
## Tasks
- [x] “WorkerStatus” latency metric support for all language workers.
- [x] Monitoring “WorkerStatus” latency and adding a new worker depending on "WorkerStatus" latency.
- [x] Removing existing concurrency limits.
- [x] Testing the feature by enabling on subset of the customers, arranging parameters such as polling intervals and delays in grpc channel, e.t.c.
- [x] Service bus session ordering guarantees with batch dispatch and multiple workers .
https://github.com/Azure/azure-functions-servicebus-extension/issues/94
- [ ] **[GA needed]** Check memory consumption before spawning new processes. If there is not enough space skip adding.
- [ ] Check CPU/thread starvation before spawning a new worker. For example if CPU > 80% most likely there will be delays in GRPC and we do not need to add new workers in this case.
- [ ] Add a worker only if the function app executes during the check. On app service plan other apps can affect the target apps.
- [ ] Add WorkerStatus request timeout so we would not wait until WorkerStatus request is returned.
- [ ] Mark a worker as unhealthy if it has constant GRPC delay and do not use it in the analysis. Shot down unhealthy worker if there is no active executions. Related - #8417.
- [ ] Load balancing invocation requests across all language workers. Now we send invocation request to the language workers using round robin - we can send message to particular worker and skip sending to busy workers.
- [ ] App lens detector to monitor adding new workers/request invocations by worker/grpc delays by worker.
- [ ] Investigate if language workers can report their state. For example we can use this library for node to detect blocking: https://www.npmjs.com/package/blocked
Contributor guide
Assessment
This issue has not been assessed yet.