apache / apache/dolphinscheduler
[DSIP-29][Task] Deprecate DISPATCH status in task instance lifecycle
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 29
Description
### Search before asking
- [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
### Motivation
Once a TaskInstance is triggered, the status will be changed follow: submit -> dispatch -> running.
Submit means the task instance is initialized, dispatch means the task has been dispatch to worker, running means the task is running in worker.
The dispatch status is useful in the history design, since `dispatch` is a async operation, but in the latest implementation, the dispatch operation is a sync operation, maintain a dispatch will bring some problems.
1. When the task is in dispatch status, we cannot track it, we don't know whether the task is died since it done's initialize task instance log, we even donot know when the task will be executed.
2. When we want to kill a task instance in worker, it's hard to do kill operation, since the task may in waiting queue or in thread pool queue.
If we deprecated the dispatch status, things will be easy, once a task has dispatch to worker we can directly set the task instance status to running. This means we will also deprecate the waiting queue in worker, since the waiting queue doesn't help to manage the task, it just make the task lifecycle more difficult and make the resource management difficult, we hope the waiting task is all manage by master, once the task has been sent to worker, it will get thread to execute.
### Design Detail
1. Remove the Dispatch status, once the task has been sent to worker then set task instance status to running.
3. Deprecate the `TaskExecuteThreadsFullPolicy` in worker, the worker will reject the task once the worker thread pool is full.
### Compatibility, Deprecation, and Migration Plan
This is compatibility, since the dispatch status will keep in code but only set it is depracated, will remove this after 3.3.0.
### Test Plan
Tested by UT and E2E
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.