Azure / Azure/azure-functions-durable-js

Failed to run n sub-orchestrators in parallel

Open
#494 23 comments 0 reactions 0 assignees View on GitHub
Needs: Investigation :mag: P1 Performance
Dominant language
TypeScript
Stars
142
Forks
66
Avg merge
3d 19h
Merged PRs (30d)
4

Description

Hello!

I have developed an ETL service using Azure Durable Functions. The service includes:
- A main function (Starter)
- A main orchestrator
- An etlOrchestrator
- Activity functions for reading, transforming, and writing data

The workflow is as follows:
- The main function is time-triggered and starts the main orchestrator
- The main orchestrator:
- Calls an activity function that returns the number of data to be processed
- Creates a matrix of etlOrchestrators, with each group containing 50 etlOrchestrators
- Calls each array of etlOrchestrators (using task.all) sequentially
- The etlOrchestrator:
- Calls the reader activity
- Calls the transformer activity
- Calls the writer activity

The main orchestrator creates three groups of etlOrchestrators. When I start the service, it successfully runs two groups of etlOrchestrators. However, for the third group, the latency increases and I receive numerous errors.:

- System.ArgumentException at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException
- Exception binding parameter 'context', An item with the same key has already been added. Key: ExternalDurablePowerShellSDK
- Transport endpoint is not connected : '/home/site/wwwroot'
- Orchestrator function 'etl-orchestrator' failed: Cannot find module './json'
- Exception while executing function: Functions.data-writer Did not find any initialized language workers

These errors are randomly, but every time when the 3rd group needs to be run the latency increases and errors occur.

**Code example**
```
const etlOrchestrators = [];

for (let i = 0; i < chunks.length; i += 1) {
etlOrchestrators.push(context.df.callSubOrchestrator('etl-orchestrator', chunks[i]));
}

const groupedEtlOrchestrators: Task[][] = [];

for (let i = 0; i < etlOrchestrators.length; i += ENV.DATA_ETL_CONCURRENT_SIZE) {
const slicedArray = etlOrchestrators.slice(i, i + ENV.DATA_ETL_CONCURRENT_SIZE);
groupedEtlOrchestrators.push(slicedArray);
}
```

**Host.json**
```
{
"version": "2.0",
"logging": {
"console": {
"isEnabled": true,
"DisableColors": false
},
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.*, 4.0.0)"
},
"extensions": {
"durableTask": {
"tracing": {
"traceInputsAndOutputs": false,
"traceReplayEvents": false
},
"maxConcurrentActivityFunctions": 20,
"maxConcurrentOrchestratorFunctions": 10
}
}
}
```

Memory/CPU used
![image](https://user-images.githubusercontent.com/37120514/226765212-008cd08b-d7df-4c51-8a8c-a0f06fa3384d.png)

**Investigative information**
"@azure/functions": "^3.5.0"
"durable-functions": "^2.1.1"
Node.js version: 16.10
Language: typescript
Runtime version ~4

***If deployed to Azure App Service***
- Function App name: dev-vehicleservice-func
- Function name(s): etl-orchestrator
- Region: West Europe
- Orchestration instance ID(s): 16a00fc5d0cc49458370f6e9f93074cc:1-147 (latest completed: 2023-03-21 22:58:39.140 16a00fc5d0cc49458370f6e9f93074cc:54 )
- Maximum Burst: 10 instances
- Minimum: 1 instance

**Additional context**
- Function invocation IDs:
9b1f55782edbffae1ff12265c9193ef2
31ab11bc371e2b144c5dabeeb38fe1d5
5ab347356e5ff6fb5a74e6b5e64910db
3be648690fafc465922748c7a1c2d285
0cad7403579881a8beab3f161e10a727
30e132f37492206b76d140471c953f52
007544fc68265fa0c15a8dec8ae8859c
b7dcd063e130ff1913dbbab73acf3285
efdb9b9533c74406e2be200dad17c7d8

Note: As this is a development environment, I occasionally delete tables, queues, and containers (excluding those ending in '-leases' and '-applease') to clear the history and re-run the service.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the provided Durable Functions orchestration code and host.json settings alongside the reported worker, transport, duplicate-key, and missing-module errors. Compare the @azure/functions and durable-functions versions with the Node.js 16 and Functions runtime configuration, then use the supplied invocation IDs and orchestration history to identify a reproducible failure. Done means the third group runs without the reported errors, or the issue is narrowed to a specific runtime or configuration defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, node.js, typescript
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.