apache / apache/dolphinscheduler
[Bug] [Worker] File parameter transfer feature is broken
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 29
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
File parameter transfer feature is completely broken in DolphinScheduler 3.4.1. The upstream task generates files with OUT type FILE parameters, but the downstream task cannot receive the files. The `resourceItemMap={}` in logs indicates that no files were downloaded to the downstream task.
After deep investigation, I found that **commit `6f12d1ded9` ([Fix-17184] Fix varpool cannot use #17199) deleted the `TaskFilesTransferUtils.java`**, which contains the critical file transfer logic:
- `uploadOutputFiles()` - uploads output files to S3 (DATA_TRANSFER directory)
- `downloadUpstreamFiles()` - downloads upstream files from S3 to local
Without these functions, files remain only in the local task working directory and downstream tasks cannot access them.
### What you expected to happen
The file parameter transfer should work as documented:
1. Upstream task (ot) generates files and declares FILE type OUT parameters
2. System automatically uploads files to S3 (DATA_TRANSFER directory)
3. Downstream task (n2) declares FILE type IN parameters with reference ot.dir-data
4. System automatically downloads files from S3 to n2's working directory before execution
5. n2 can access the files locally
### How to reproduce
1. Create a workflow with two Shell tasks (ot -> n2)
2. Configure ot task:
- rawScript: `mkdir -p data/test1 data/test2 && echo "test1 message" >> data/test1/text.txt && echo "test2 message" >> data/test2/text.txt && tree .`
- localParams: `dir-data` (OUT, FILE, value=data)
3. Configure n2 task:
- rawScript: `tree . && cat input_dir/test1/text.txt && cat input_dir/test2/text.txt`
- localParams: `input_dir` (IN, FILE, value=ot.dir-data)
4. Execute workflow
5. Result: n2 fails with "No such file or directory"
### Anything else
**MinIO bucket verification (empty):**
```bash
$ mc ls myminio/dolphinscheduler/
[2026-04-10 11:40:25 CST] 0B dolphinscheduler/
```
**No DATA_TRANSFER directory - files were never uploaded.**
**ot task (upstream) logs:**
```
Initialize shell task params {
"localParams" : [ {
"prop" : "dir-data",
"direct" : "OUT",
"type" : "FILE",
"value" : "data"
} ],
"varPool" : [ ],
...
}
Set taskVarPool: null successfully
```
**n2 task (downstream) logs:**
```
Download resources successfully: ResourceContext(resourceItemMap={})
Initialize shell task params {
"localParams" : [ {
"prop" : "input_dir",
"direct" : "IN",
"type" : "FILE",
"value" : "ot.dir-data"
} ],
...
}
cat: input_dir/test1/text.txt: No such file or directory
```
### Version
3.4.1
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by tracing the worker's file-parameter transfer flow and the deleted TaskFilesTransferUtils.java, especially uploadOutputFiles() and downloadUpstreamFiles(). Reproduce the two-Shell-task workflow from the issue, then verify that output files reach the S3 DATA_TRANSFER directory and are available in the downstream task's working directory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100