conductor-oss / conductor-oss/conductor
Duplicate task execution when system task produce large outputs
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d
- Merged PRs (30d)
- 33
Description
**Describe the bug**
When a task such as "INLINE" or "JSON_JQ_TRANSFORM" produces a large output, it can result in duplicate tasks being run, all on attempt 0.
These duplicate tasks, when finished, can then cause subsequent tasks to also be executed multiple times.
I have not tested this with tasks other than 'INLINE' and 'JSON_JQ_TRANSFORM', so it may also affect other system tasks.
Example workflow:
``` json
{
"name": "error_workflow",
"description": "",
"version": 1,
"tasks": [
{
"name": "do_while_loop",
"taskReferenceName": "do_while_loop",
"type": "DO_WHILE",
"loopCondition": "if ($.do_while_loop.iteration < 200) { true; } else { false; }",
"loopOver": [
{
"name": "inline",
"taskReferenceName": "inline_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": "${do_while_loop.output.iteration}"
}
},
{
"name": "inline",
"taskReferenceName": "inline_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
}
],
"inputParameters": {}
}
],
"inputParameters": [],
"outputParameters": {},
"schemaVersion": 2
}
```
This issue can also occur without using a "DO_WHILE" loop, simply by having a workflow with around 200 tasks that each produce large outputs.
Example:
``` json
{
"name": "do_while_jq_simple_1000",
"description": "",
"version": 1,
"tasks": [
{
"name": "inline",
"taskReferenceName": "inline_0_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": 1
}
},
{
"name": "inline",
"taskReferenceName": "inline_0_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
},
{
"name": "inline",
"taskReferenceName": "inline_1_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": 1
}
},
{
"name": "inline",
"taskReferenceName": "inline_1_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
},
...
{
"name": "inline",
"taskReferenceName": "inline_199_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": 1
}
},
{
"name": "inline",
"taskReferenceName": "inline_199_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
}
],
"inputParameters": [],
"outputParameters": {},
"schemaVersion": 2
}
```
UI output just using `"INLINE"` tasks

UI output with "`DO_WHILE`"

Attempt in UI

**Details**
Conductor version: 3.x
Persistence implementation: Postgres
Queue implementation: Redis
Lock: Redis
**To Reproduce**
1. Create the example workflow.
2. Run the workflow.
3. Inspect the tasks in the workflow later tasks will have duplicate runs.
**Additional context**
We've got multiple instances running one Conductor with the config:
```
conductor.system-task-workers.enabled=false
conductor.workflow-repair-service.enabled=true
```
and have 4 instances with:
```
conductor.system-task-workers.enabled=true
conductor.workflow-repair-service.enabled=false
```
I found this bug when testing https://github.com/conductor-oss/conductor/issues/604
Contributor guide
Assessment
This issue has not been assessed yet.