apache / apache/dolphinscheduler

[Bug] [Worker] File parameter transfer feature is broken

Closed
#18,166 8 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Java
Stars
14.5k
Forks
5.1k
Avg merge
1d 21h
Merged PRs (30d)
29

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.

### What happened

File parameter transfer feature is completely broken in DolphinScheduler 3.4.1. The upstream task generates files with OUT type FILE parameters, but the downstream task cannot receive the files. The `resourceItemMap={}` in logs indicates that no files were downloaded to the downstream task.

After deep investigation, I found that **commit `6f12d1ded9` ([Fix-17184] Fix varpool cannot use #17199) deleted the `TaskFilesTransferUtils.java`**, which contains the critical file transfer logic:
- `uploadOutputFiles()` - uploads output files to S3 (DATA_TRANSFER directory)
- `downloadUpstreamFiles()` - downloads upstream files from S3 to local

Without these functions, files remain only in the local task working directory and downstream tasks cannot access them.

### What you expected to happen

The file parameter transfer should work as documented:

1. Upstream task (ot) generates files and declares FILE type OUT parameters
2. System automatically uploads files to S3 (DATA_TRANSFER directory)
3. Downstream task (n2) declares FILE type IN parameters with reference ot.dir-data
4. System automatically downloads files from S3 to n2's working directory before execution
5. n2 can access the files locally

### How to reproduce

1. Create a workflow with two Shell tasks (ot -> n2)
2. Configure ot task:
- rawScript: `mkdir -p data/test1 data/test2 && echo "test1 message" >> data/test1/text.txt && echo "test2 message" >> data/test2/text.txt && tree .`
- localParams: `dir-data` (OUT, FILE, value=data)
3. Configure n2 task:
- rawScript: `tree . && cat input_dir/test1/text.txt && cat input_dir/test2/text.txt`
- localParams: `input_dir` (IN, FILE, value=ot.dir-data)
4. Execute workflow
5. Result: n2 fails with "No such file or directory"

### Anything else

**MinIO bucket verification (empty):**
```bash
$ mc ls myminio/dolphinscheduler/
[2026-04-10 11:40:25 CST] 0B dolphinscheduler/
```
**No DATA_TRANSFER directory - files were never uploaded.**

**ot task (upstream) logs:**
```
Initialize shell task params {
"localParams" : [ {
"prop" : "dir-data",
"direct" : "OUT",
"type" : "FILE",
"value" : "data"
} ],
"varPool" : [ ],
...
}
Set taskVarPool: null successfully
```

**n2 task (downstream) logs:**
```
Download resources successfully: ResourceContext(resourceItemMap={})
Initialize shell task params {
"localParams" : [ {
"prop" : "input_dir",
"direct" : "IN",
"type" : "FILE",
"value" : "ot.dir-data"
} ],
...
}
cat: input_dir/test1/text.txt: No such file or directory
```

### Version

3.4.1

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the worker's file-parameter transfer flow and the deleted TaskFilesTransferUtils.java, especially uploadOutputFiles() and downloadUpstreamFiles(). Reproduce the two-Shell-task workflow from the issue, then verify that output files reach the S3 DATA_TRANSFER directory and are available in the downstream task's working directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.