conductor-oss / conductor-oss/conductor

Duplicate task execution when system task produce large outputs

Open
#613 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d
Merged PRs (30d)
33

Description

**Describe the bug**
When a task such as "INLINE" or "JSON_JQ_TRANSFORM" produces a large output, it can result in duplicate tasks being run, all on attempt 0.
These duplicate tasks, when finished, can then cause subsequent tasks to also be executed multiple times.
I have not tested this with tasks other than 'INLINE' and 'JSON_JQ_TRANSFORM', so it may also affect other system tasks.

Example workflow:
``` json
{
"name": "error_workflow",
"description": "",
"version": 1,
"tasks": [
{
"name": "do_while_loop",
"taskReferenceName": "do_while_loop",
"type": "DO_WHILE",
"loopCondition": "if ($.do_while_loop.iteration < 200) { true; } else { false; }",
"loopOver": [
{
"name": "inline",
"taskReferenceName": "inline_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": "${do_while_loop.output.iteration}"
}
},
{
"name": "inline",
"taskReferenceName": "inline_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
}
],
"inputParameters": {}
}
],
"inputParameters": [],
"outputParameters": {},
"schemaVersion": 2
}
```

This issue can also occur without using a "DO_WHILE" loop, simply by having a workflow with around 200 tasks that each produce large outputs.

Example:
``` json
{
"name": "do_while_jq_simple_1000",
"description": "",
"version": 1,
"tasks": [
{
"name": "inline",
"taskReferenceName": "inline_0_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": 1
}
},
{
"name": "inline",
"taskReferenceName": "inline_0_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
},
{
"name": "inline",
"taskReferenceName": "inline_1_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": 1
}
},
{
"name": "inline",
"taskReferenceName": "inline_1_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
},
...
{
"name": "inline",
"taskReferenceName": "inline_199_ref",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ var out = []; for (var i = 0; i < $.max; i++) { out.push($.max * $.key + i); } return out; })()",
"max": 10000,
"key": 1
}
},
{
"name": "inline",
"taskReferenceName": "inline_199_a",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "javascript",
"expression": "(function(){ return 1; })()"
}
}
],
"inputParameters": [],
"outputParameters": {},
"schemaVersion": 2
}
```

UI output just using `"INLINE"` tasks
![Image](https://github.com/user-attachments/assets/2a0cb7d8-d054-4239-92a7-cab63b4efbee)

UI output with "`DO_WHILE`"
![Image](https://github.com/user-attachments/assets/30cc99a3-1b40-4e63-8493-4032726a78af)

Attempt in UI
![Image](https://github.com/user-attachments/assets/fa117d1f-2eef-44b2-a2cd-73641e1f324b)

**Details**
Conductor version: 3.x
Persistence implementation: Postgres
Queue implementation: Redis
Lock: Redis

**To Reproduce**
1. Create the example workflow.
2. Run the workflow.
3. Inspect the tasks in the workflow later tasks will have duplicate runs.

**Additional context**
We've got multiple instances running one Conductor with the config:
```
conductor.system-task-workers.enabled=false
conductor.workflow-repair-service.enabled=true
```
and have 4 instances with:
```
conductor.system-task-workers.enabled=true
conductor.workflow-repair-service.enabled=false
```
I found this bug when testing https://github.com/conductor-oss/conductor/issues/604

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.