conductor-oss / conductor-oss/conductor
[FEATURE]: Support running JOIN tasks synchronously
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 37
Description
## Describe the Feature Request
We have recently tried running JOIN tasks synchronously in Conductor. To do this we made a small change to WorkflowExecutorOps.isLazyEvaluateWorkflow and updated the JOIN task to return isAsync = false.
Our reason for switching to sync are
1. predictable completion of the JOIN without the need to tune any settings
2. reduction in JOIN executions for long running JOINs
## Describe Preferred Solution
Make existing JOIN task sync
Pros
- simple change
- likely will be more performant for the majority
Cons
- maybe some use cases it results in more JOIN executions
## Describe Alternatives
1. Have the JOIN task be driven by a system property
Pros
- this is an fairly simple change
- gives consumers the option, although still a single setting for all JOINs
Cons
- may make maintenance of execution. logic more complicated
2. Have the JOIN task be either async or sync based on TaskDef
- would require isAsync to be passed the task which won’t work initializing queues
- might need to introduce a new method supportsAsync()
Pros
- this is a flexible solution allowing workflow owners to choose
Cons
- more complexity in making this change
3. Have both a sync and non sync JOIN task
- any conditional logic related to execution of JOINS will not be able to use the TaskType there will need to be another attribute of
tasks i.e. WorkerSystemTask.isJoin()
Pros
- this is a flexible solution allowing workflow owners to choose which JOIN they use
Cons
- more complexity in making this change
Contributor guide
Assessment
This issue has not been assessed yet.