apache / apache/dolphinscheduler

[WIP][DSIP-99][TaskPlugin] Save task output to a separate file

Open
#17,791 7 comments 0 reactions 0 assignees View on GitHub
discussion DSIP priority:middle
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.

### Motivation

Currently, we store the task log and task output in the same file.

The problem is that the task log is generated by `DolphinScheduler`, the end users don't actually understand the underlying logic. They are more care about the output of their tasks.

Therefore, they need to locate their task outputs within the vast array of task instances.

Additionally, since we often use separate threads to collect task outputs, this can also lead to issues where task outputs and task log entries become interspersed and out of order.

This DSIP aims to separate the task log and task output and use an extra file to store the task output.

### Design Detail

todo:

### Compatibility, Deprecation, and Migration Plan

todo:

### Test Plan

todo:

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. First review the task plugin's existing task-log and task-output handling, then clarify the design and migration plan before implementing. Done should mean task output is stored separately from DolphinScheduler logs without interleaving, with a test plan covering the new behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.