apache / apache/hop

[Bug]: Move files - can't process files using previous results as args

Open
#4,045 0 comments 0 reactions 0 assignees View on GitHub
Actions awaiting triage P2 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### Apache Hop version?

2.9.0

### Java version?

openjdk version "21.0.2" 2024-01-16

### Operating system

Linux

### What happened?

Steps to reproduce:
- create a pipeline that generates a recordset with the fields requested by the Move files action: you can simply add a Data grid (with the fields src_path, dst_path and wildcard, and respective values) and link it with a Copy rows to result
- create a workflow that uses the above pipeline and link it to a Move files action
- in the Move files action, check "Copy previous results to args" and "Destination is a file"

When dealing with local files, the action fails with the following error:
```
2024/06/12 10:01:33 - Move files - found [1] argument(s) from previous result
2024/06/12 10:01:33 - Move files - Processing row source File/folder source : [/work/hop/mount/iraiser/in/expIraiser_2024_06_07.csv] ... destination file/folder : [/work/hop/mount/iraiser/in/_deleteme_expIraiser_2024_06_07.csv]... wildcard : [null]
2024/06/12 10:01:33 - Move files - ERROR: Can not move file/folder [/work/hop/mount/iraiser/in/expIraiser_2024_06_07.csv] to [file:///work/hop/mount/iraiser/in/_deleteme_expIraiser_2024_06_07.csv]. Exception : [
2024/06/12 10:01:33 - Move files - Could not extract the file name from the URI: file:///work/hop/mount/iraiser/in/_deleteme_expIraiser_2024_06_07.csv
2024/06/12 10:01:33 - Move files - ]
2024/06/12 10:01:33 - Move files - =======================================
2024/06/12 10:01:33 - Move files - Total files in error : 1
2024/06/12 10:01:33 - Move files - Total files successfully processed : 0
2024/06/12 10:01:33 - Move files - =======================================
2024/06/12 10:01:33 - _stub-move-files - Finished action [Move files] (result=[false])
```

When dealing with remote files (like those in an sFTP server), Hop throws an Exception: it cannot print an error because a variable `destinationfilefolder` is null (see #3985 for the log trace).

In both cases, the Detailed log (example above) shows that the file names and the (empty) wildcard are correctly passed from the pipeline to the Move files action

### Issue Priority

Priority: 2

### Issue Component

Component: Actions, Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the workflow with a Data grid, Copy rows to result, and the Move files action, testing both local and remote files. Trace how the Move files action handles previous-result arguments when "Destination is a file" is selected, including the URI and null-destination errors described here and in #3985. Done means both local and remote moves complete without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.