apache / apache/dolphinscheduler

[DSIP-69] Fix master dispatch task timeout might cause task duplicate running in worker

Open
#16,481 0 comments 1 reaction 1 assignee Claimed by @ruanwenjun View on GitHub
DSIP priority:middle
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
28

Description

### Search before asking

- [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

Right now, there exist some case might cause the task duplicated dispatched.
e.g.

image
The master dispatch task a to worker A first, but receive a timeout response, this might happen when the worker rpc is busy, then master will select a new worker B and retry the dispatch.

Then there might exist two situations:
1. The task has been received by worker A, then take will duplicate exist in worker A and worker B, both the two worker will execute the task, a worser case is the task might duplicated in more worker.
2. The task hasn't been received by worker A, then task will not duplicate executed.

The first situation is not accepted.

### Design Detail

In order to solve this, we should change the dispatch logic.

image

### Compatibility, Deprecation, and Migration Plan

_No response_

### Test Plan

_No response_

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.