apache / apache/dolphinscheduler
[Bug] [DAO] findLastTaskInstances returns duplicated task instances and breaks the dependent task
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 29
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
`TaskInstanceMapper#findLastTaskInstances` is supposed to return **the last task instance
per task code** of a workflow instance. It resolves "last" by joining on the maximum
`end_time`:
```xml
from t_ds_task_instance instance
join (
select task_code, max(end_time) as max_end_time, workflow_instance_id
from t_ds_task_instance
where 1=1
and workflow_instance_id = #{workflowInstanceId}
and state != 8
...
group by task_code
) t_max
on instance.workflow_instance_id = t_max.workflow_instance_id
and instance.task_code = t_max.task_code
and instance.end_time = t_max.max_end_time
```
`end_time` is not unique. When two attempts of the same task share the very same
`end_time`, both of them match `instance.end_time = t_max.max_end_time` and the query
returns **two rows for one task code**.
This is easy to hit on MySQL, where `t_ds_task_instance.end_time` is declared as
```sql
`end_time` datetime DEFAULT NULL COMMENT 'task end time',
```
a `datetime` **without fractional seconds**, so every timestamp is truncated to a whole
second. A task that fails and is retried quickly (a short or zero failed-retry interval,
or simply a fast-failing task) ends up with the failed attempt and the retry sharing an
identical `end_time`. Note the query filters on `state != 8` only — it does not filter on
`flag`, so the invalidated previous attempts are included as well.
The only caller is `DependentExecute#dependResultByAllTaskOfWorkflowInstance`, which keys
the result by task code:
```java
Map taskExecutionStatusMap =
taskInstanceList.stream()
.filter(taskInstance -> taskInstance.getTaskExecuteType() != TaskExecuteType.STREAM)
.collect(Collectors.toMap(TaskInstance::getTaskCode, TaskInstance::getState));
```
`Collectors.toMap` has no merge function, so a duplicated task code throws
```
java.lang.IllegalStateException: Duplicate key (attempted merging values ... and ...)
```
and the dependency evaluation of the DEPENDENT task blows up.
### What you expected to happen
`findLastTaskInstances` returns at most one task instance per task code — the last
attempt — so a DEPENDENT task can evaluate the upstream workflow normally.
### How to reproduce
1. Workflow **A** contains a task that fails and then succeeds on retry, with a short
failed-retry interval so that both attempts finish within the same second (on MySQL
any two attempts finishing in the same second are enough).
2. Workflow **B** contains a DEPENDENT task depending on workflow **A** with
**"ALL tasks"** selected (`DEPENDENT_ALL_TASK_CODE`).
3. Run A to completion, then run B.
4. B's dependent check fails with `IllegalStateException: Duplicate key` instead of
resolving the dependency.
It also reproduces directly at the DAO level — insert two task instances with the same
`task_code` and the same `end_time` into one workflow instance and call
`queryLastTaskInstanceListIntervalInWorkflowInstance`; it returns 2 rows instead of 1.
### Anything else
The same duplication would silently affect any future caller of this query, since the
method name (`findLastTaskInstances` / `queryLastTaskInstanceListIntervalInWorkflowInstance`)
promises one row per task code.
### Version
dev
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start with TaskInstanceMapper#findLastTaskInstances and the query named queryLastTaskInstanceListIntervalInWorkflowInstance; inspect how the maximum end_time is selected and read its only caller, DependentExecute#dependResultByAllTaskOfWorkflowInstance. Reproduce the DAO case with two task instances sharing task_code and end_time, then verify that the dependent workflow completes without a duplicate-key failure and only one instance is returned per task code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100