gcsfs put copies files out of order
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
i am trying to copy multiple files using gcsfs put API. The syntax provided to copy multiple files goes like this. lpath1 is supposed to put the data into rapth1 and respectively for other files.
gcs.put([lpath1, lpath2, lpath3, lapth4], [rpath1, rpath2, rpath3, rpath4])
Issue:
The copy happens out of order. lpath1 write the results to any of the four output paths, and so are the other input files. Is this behavior expected.
Expected Behavior:
lpath1 is supposed to put the data into rapth1 and respectively for other files
lpath1->rpath1
lpath2->rpath2
lpath3->rpath3
lpath4->rpath4
Example:
fs.put(
[f'{local_path}/f' for f in files] ,
[f'{gcs_path}/f' for f in files],
recursive=True
)
ls -lh on Local files give:
-rw-r--r--. 1 root root 169M Mar 25 15:37 01-080000
-rw-r--r--. 1 root root 179M Mar 25 15:37 01-090000
-rw-r--r--. 1 root root 173M Mar 25 15:37 01-100000
-rw-r--r--. 1 root root 202M Mar 25 15:37 01-110000
After the copy using above command, ls -lh on gcp buckets gives:
172.61 MiB 2022-03-25T15:38:15Z gs:///01-080000
TOTAL: 1 objects, 180989509 bytes (172.61 MiB)
201.31 MiB 2022-03-25T15:38:15Z gs:///01-090000
TOTAL: 1 objects, 211084829 bytes (201.31 MiB)
168.31 MiB 2022-03-25T15:38:16Z gs:///01-100000
TOTAL: 1 objects, 176486219 bytes (168.31 MiB)
178.05 MiB 2022-03-25T15:38:15Z gs:///01-110000
TOTAL: 1 objects, 186697415 bytes (178.05 MiB)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the shown fs.put call and paired local and GCS paths, then inspect the put entry point and its handling of multiple paths. Done means each local path is written to the corresponding remote path, with a regression test covering the mapping.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100