conductor-oss / conductor-oss/conductor

[FEATURE]: Support running JOIN tasks synchronously

Open
#387 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 3h
Merged PRs (30d)
37

Description

## Describe the Feature Request

We have recently tried running JOIN tasks synchronously in Conductor. To do this we made a small change to WorkflowExecutorOps.isLazyEvaluateWorkflow and updated the JOIN task to return isAsync = false.

Our reason for switching to sync are
1. predictable completion of the JOIN without the need to tune any settings
2. reduction in JOIN executions for long running JOINs

## Describe Preferred Solution

Make existing JOIN task sync
Pros
- simple change
- likely will be more performant for the majority
Cons
- maybe some use cases it results in more JOIN executions

## Describe Alternatives

1. Have the JOIN task be driven by a system property
Pros
- this is an fairly simple change
- gives consumers the option, although still a single setting for all JOINs
Cons
- may make maintenance of execution. logic more complicated

2. Have the JOIN task be either async or sync based on TaskDef
- would require isAsync to be passed the task which won’t work initializing queues
- might need to introduce a new method supportsAsync()
Pros
- this is a flexible solution allowing workflow owners to choose
Cons
- more complexity in making this change

3. Have both a sync and non sync JOIN task
- any conditional logic related to execution of JOINS will not be able to use the TaskType there will need to be another attribute of
tasks i.e. WorkerSystemTask.isJoin()
Pros
- this is a flexible solution allowing workflow owners to choose which JOIN they use
Cons
- more complexity in making this change

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.