apache / apache/dolphinscheduler

[DSIP-49][Workflow] Rerun across workflow or project with dependency automatically

Open
#16,194 0 comments 0 reactions 1 assignee Claimed by @zhuxt2015 View on GitHub
DSIP
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

There are three workflow in the one project ,such as ods,dwd,dws. in workflow ods, it include two task A,B. in workflow dwd,it include two task C,D. in workflow dws ,it include one task E. one of the dependency relationship describe as follow:
A->C->E
If the calculation logic of task A is wrong or the task failed, the downstream tasks C and E need to be rerun, and A, C and E can only be rerun manually, instead of automatically starting C and E when A finished.

### Design Detail

Through t_ds_process_task_relation, t_ds_process_definition, t_ds_task_definition three tables can get the task lineage, then all downstream tasks of task A can be found, Then we can rerun the task layer by layer according to the dependency hierarchy. for example, the downstream of A has task C, and the downstream of C is E, then when A completes the rerun, start to rerun C, wait for C to run completely, and start to rerun E.

### Compatibility, Deprecation, and Migration Plan

1. If there is a running instance of the rerun task, you need to close it first to avoid unnecessary errors caused by running the same task instance at the same time.
2. You can only rerun the workflow of your own execution permission

### Test Plan

_No response_

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.