Azure / Azure/azure-functions-durable-js
Failed to run n sub-orchestrators in parallel
- Dominant language
- TypeScript
- Stars
- 142
- Forks
- 66
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 4
Description
Hello!
I have developed an ETL service using Azure Durable Functions. The service includes:
- A main function (Starter)
- A main orchestrator
- An etlOrchestrator
- Activity functions for reading, transforming, and writing data
The workflow is as follows:
- The main function is time-triggered and starts the main orchestrator
- The main orchestrator:
- Calls an activity function that returns the number of data to be processed
- Creates a matrix of etlOrchestrators, with each group containing 50 etlOrchestrators
- Calls each array of etlOrchestrators (using task.all) sequentially
- The etlOrchestrator:
- Calls the reader activity
- Calls the transformer activity
- Calls the writer activity
The main orchestrator creates three groups of etlOrchestrators. When I start the service, it successfully runs two groups of etlOrchestrators. However, for the third group, the latency increases and I receive numerous errors.:
- System.ArgumentException at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException
- Exception binding parameter 'context', An item with the same key has already been added. Key: ExternalDurablePowerShellSDK
- Transport endpoint is not connected : '/home/site/wwwroot'
- Orchestrator function 'etl-orchestrator' failed: Cannot find module './json'
- Exception while executing function: Functions.data-writer Did not find any initialized language workers
These errors are randomly, but every time when the 3rd group needs to be run the latency increases and errors occur.
**Code example**
```
const etlOrchestrators = [];
for (let i = 0; i < chunks.length; i += 1) {
etlOrchestrators.push(context.df.callSubOrchestrator('etl-orchestrator', chunks[i]));
}
const groupedEtlOrchestrators: Task[][] = [];
for (let i = 0; i < etlOrchestrators.length; i += ENV.DATA_ETL_CONCURRENT_SIZE) {
const slicedArray = etlOrchestrators.slice(i, i + ENV.DATA_ETL_CONCURRENT_SIZE);
groupedEtlOrchestrators.push(slicedArray);
}
```
**Host.json**
```
{
"version": "2.0",
"logging": {
"console": {
"isEnabled": true,
"DisableColors": false
},
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.*, 4.0.0)"
},
"extensions": {
"durableTask": {
"tracing": {
"traceInputsAndOutputs": false,
"traceReplayEvents": false
},
"maxConcurrentActivityFunctions": 20,
"maxConcurrentOrchestratorFunctions": 10
}
}
}
```
Memory/CPU used

**Investigative information**
"@azure/functions": "^3.5.0"
"durable-functions": "^2.1.1"
Node.js version: 16.10
Language: typescript
Runtime version ~4
***If deployed to Azure App Service***
- Function App name: dev-vehicleservice-func
- Function name(s): etl-orchestrator
- Region: West Europe
- Orchestration instance ID(s): 16a00fc5d0cc49458370f6e9f93074cc:1-147 (latest completed: 2023-03-21 22:58:39.140 16a00fc5d0cc49458370f6e9f93074cc:54 )
- Maximum Burst: 10 instances
- Minimum: 1 instance
**Additional context**
- Function invocation IDs:
9b1f55782edbffae1ff12265c9193ef2
31ab11bc371e2b144c5dabeeb38fe1d5
5ab347356e5ff6fb5a74e6b5e64910db
3be648690fafc465922748c7a1c2d285
0cad7403579881a8beab3f161e10a727
30e132f37492206b76d140471c953f52
007544fc68265fa0c15a8dec8ae8859c
b7dcd063e130ff1913dbbab73acf3285
efdb9b9533c74406e2be200dad17c7d8
Note: As this is a development environment, I occasionally delete tables, queues, and containers (excluding those ending in '-leases' and '-applease') to clear the history and re-run the service.
Contributor guide
Research direction
Start by reviewing the provided Durable Functions orchestration code and host.json settings alongside the reported worker, transport, duplicate-key, and missing-module errors. Compare the @azure/functions and durable-functions versions with the Node.js 16 and Functions runtime configuration, then use the supplied invocation IDs and orchestration history to identify a reproducible failure. Done means the third group runs without the reported errors, or the issue is narrowed to a specific runtime or configuration defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, node.js, typescript
- Domain
- backend, cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100