apache / apache/dolphinscheduler
[WIP][DSIP-99][TaskPlugin] Save task output to a separate file
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 29
Description
### Search before asking
- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
### Motivation
Currently, we store the task log and task output in the same file.
The problem is that the task log is generated by `DolphinScheduler`, the end users don't actually understand the underlying logic. They are more care about the output of their tasks.
Therefore, they need to locate their task outputs within the vast array of task instances.
Additionally, since we often use separate threads to collect task outputs, this can also lead to issues where task outputs and task log entries become interspersed and out of order.
This DSIP aims to separate the task log and task output and use an extra file to store the task output.
### Design Detail
todo:
### Compatibility, Deprecation, and Migration Plan
todo:
### Test Plan
todo:
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
No source files, tests, or entry points are named. First review the task plugin's existing task-log and task-output handling, then clarify the design and migration plan before implementing. Done should mean task output is stored separately from DolphinScheduler logs without interleaving, with a test plan covering the new behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100