apache / apache/dolphinscheduler

[Bug] [DAO] findLastTaskInstances returns duplicated task instances and breaks the dependent task

Closed
#18,543 3 comments 0 reactions 0 assignees View on GitHub
bug discussion
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.

### What happened

`TaskInstanceMapper#findLastTaskInstances` is supposed to return **the last task instance
per task code** of a workflow instance. It resolves "last" by joining on the maximum
`end_time`:

```xml
from t_ds_task_instance instance
join (
select task_code, max(end_time) as max_end_time, workflow_instance_id
from t_ds_task_instance
where 1=1
and workflow_instance_id = #{workflowInstanceId}
and state != 8
...
group by task_code
) t_max
on instance.workflow_instance_id = t_max.workflow_instance_id
and instance.task_code = t_max.task_code
and instance.end_time = t_max.max_end_time
```

`end_time` is not unique. When two attempts of the same task share the very same
`end_time`, both of them match `instance.end_time = t_max.max_end_time` and the query
returns **two rows for one task code**.

This is easy to hit on MySQL, where `t_ds_task_instance.end_time` is declared as

```sql
`end_time` datetime DEFAULT NULL COMMENT 'task end time',
```

a `datetime` **without fractional seconds**, so every timestamp is truncated to a whole
second. A task that fails and is retried quickly (a short or zero failed-retry interval,
or simply a fast-failing task) ends up with the failed attempt and the retry sharing an
identical `end_time`. Note the query filters on `state != 8` only — it does not filter on
`flag`, so the invalidated previous attempts are included as well.

The only caller is `DependentExecute#dependResultByAllTaskOfWorkflowInstance`, which keys
the result by task code:

```java
Map taskExecutionStatusMap =
taskInstanceList.stream()
.filter(taskInstance -> taskInstance.getTaskExecuteType() != TaskExecuteType.STREAM)
.collect(Collectors.toMap(TaskInstance::getTaskCode, TaskInstance::getState));
```

`Collectors.toMap` has no merge function, so a duplicated task code throws

```
java.lang.IllegalStateException: Duplicate key (attempted merging values ... and ...)
```

and the dependency evaluation of the DEPENDENT task blows up.

### What you expected to happen

`findLastTaskInstances` returns at most one task instance per task code — the last
attempt — so a DEPENDENT task can evaluate the upstream workflow normally.

### How to reproduce

1. Workflow **A** contains a task that fails and then succeeds on retry, with a short
failed-retry interval so that both attempts finish within the same second (on MySQL
any two attempts finishing in the same second are enough).
2. Workflow **B** contains a DEPENDENT task depending on workflow **A** with
**"ALL tasks"** selected (`DEPENDENT_ALL_TASK_CODE`).
3. Run A to completion, then run B.
4. B's dependent check fails with `IllegalStateException: Duplicate key` instead of
resolving the dependency.

It also reproduces directly at the DAO level — insert two task instances with the same
`task_code` and the same `end_time` into one workflow instance and call
`queryLastTaskInstanceListIntervalInWorkflowInstance`; it returns 2 rows instead of 1.

### Anything else

The same duplication would silently affect any future caller of this query, since the
method name (`findLastTaskInstances` / `queryLastTaskInstanceListIntervalInWorkflowInstance`)
promises one row per task code.

### Version

dev

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with TaskInstanceMapper#findLastTaskInstances and the query named queryLastTaskInstanceListIntervalInWorkflowInstance; inspect how the maximum end_time is selected and read its only caller, DependentExecute#dependResultByAllTaskOfWorkflowInstance. Reproduce the DAO case with two task instances sharing task_code and end_time, then verify that the dependent workflow completes without a duplicate-key failure and only one instance is returned per task code.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.