apache / apache/dolphinscheduler

[DSIP-29][Task] Deprecate DISPATCH status in task instance lifecycle

Open
#15,882 2 comments 0 reactions 1 assignee Claimed by @ruanwenjun View on GitHub
DSIP
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

Once a TaskInstance is triggered, the status will be changed follow: submit -> dispatch -> running.
Submit means the task instance is initialized, dispatch means the task has been dispatch to worker, running means the task is running in worker.

image

The dispatch status is useful in the history design, since `dispatch` is a async operation, but in the latest implementation, the dispatch operation is a sync operation, maintain a dispatch will bring some problems.

1. When the task is in dispatch status, we cannot track it, we don't know whether the task is died since it done's initialize task instance log, we even donot know when the task will be executed.
2. When we want to kill a task instance in worker, it's hard to do kill operation, since the task may in waiting queue or in thread pool queue.

If we deprecated the dispatch status, things will be easy, once a task has dispatch to worker we can directly set the task instance status to running. This means we will also deprecate the waiting queue in worker, since the waiting queue doesn't help to manage the task, it just make the task lifecycle more difficult and make the resource management difficult, we hope the waiting task is all manage by master, once the task has been sent to worker, it will get thread to execute.

### Design Detail

1. Remove the Dispatch status, once the task has been sent to worker then set task instance status to running.

image

3. Deprecate the `TaskExecuteThreadsFullPolicy` in worker, the worker will reject the task once the worker thread pool is full.

### Compatibility, Deprecation, and Migration Plan

This is compatibility, since the dispatch status will keep in code but only set it is depracated, will remove this after 3.3.0.

### Test Plan

Tested by UT and E2E

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.