[INFO] Task scheduling behavior with multiple processes

I would like to clarify my understanding of how the azure functions host dispatches tasks to multiple processes.

I currently have a Python function app with the following attributes:

- Runs as a Docker container on a premium App Service Plan (P3V3 - 8vCPU, 32GB RAM)
- Functions are triggered from Azure storage account queues
- Multiple functions with a mix of IO and CPU bound work with an expected run time of 1-2 minutes per message
- `FUNCTIONS_WORKER_PROCESS_COUNT` set to 8
- Host file sets `batchSize` to 1 and `newBatchThreshold` to 7 which should give 8 concurrent messages being processed
- My app service plan is set to scale out to 30 nodes based on queue length

As I understand it, when a new host is added by the scaler it will start with a single worker process and add a new process up to my limit (8) over time (every 15 seconds?). The task dispatcher will distribute the messages from the queue over the available processes in a round robin fashion.

Based on this, when a new host starts, I would see all 8 concurrent messages dispatched to the same worker process (as the others haven't started yet). As messages are completed, new messages would start being dispatched to the other processes.

Does this all sound correct? I added a log message at the start of my process to report the process ID, and the logs from App Insights seem to corroborate the above:

Assuming this is correct, is there a way to prevent the dispatched from dispatching all 8 messages to the first process prior to the others starting?

The impact I am seeing is all 8 messages are contesting for a single CPU resource on the same process. This increases processing time for those initially messages considerably, but also causes network timeout issues with the IO bound parts of the code as they are being blocked by other CPU bound tasks on the same process.

The only work around I have thought of since encountering this issue seems extreme but is as follows:

- Deploy the function app to Azure Kubernetes Service instead
- Limit the app to a single process and a single concurrent message
- Schedule a pod per vCPU e.g. with a host running 8vCPU, I would end up with 8 pods each processing 1 concurrent message

As mentioned, this seems a bit extreme and inefficient, but I'm not sure how else I can avoid this startup behavior?

One other thing to clarify: triggering functions from an Azure storage queue manages the concurrent messages on a per queue/function basis, so each queue/function will have up to 8 concurrent messages. Is this correct? Is there a way to manage the concurrency across all queues, or would I need to move to a Service Bus queue trigger (which I believe has global concurrency across all queues)?

Additionally, how does the service bus queue trigger choose which queue to dispatch from next if the concurrency is managed globally?