fsspec / fsspec/gcsfs

gcsfs put copies files out of order

Open
#468 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

i am trying to copy multiple files using gcsfs put API. The syntax provided to copy multiple files goes like this. lpath1 is supposed to put the data into rapth1 and respectively for other files.
gcs.put([lpath1, lpath2, lpath3, lapth4], [rpath1, rpath2, rpath3, rpath4])

Issue:
The copy happens out of order. lpath1 write the results to any of the four output paths, and so are the other input files. Is this behavior expected.

Expected Behavior:
lpath1 is supposed to put the data into rapth1 and respectively for other files
lpath1->rpath1
lpath2->rpath2
lpath3->rpath3
lpath4->rpath4

Example:
fs.put(
[f'{local_path}/f' for f in files] ,
[f'{gcs_path}/f' for f in files],
recursive=True
)

ls -lh on Local files give:

-rw-r--r--. 1 root root 169M Mar 25 15:37 01-080000
-rw-r--r--. 1 root root 179M Mar 25 15:37 01-090000
-rw-r--r--. 1 root root 173M Mar 25 15:37 01-100000
-rw-r--r--. 1 root root 202M Mar 25 15:37 01-110000

After the copy using above command, ls -lh on gcp buckets gives:
172.61 MiB 2022-03-25T15:38:15Z gs:///01-080000
TOTAL: 1 objects, 180989509 bytes (172.61 MiB)
201.31 MiB 2022-03-25T15:38:15Z gs:///01-090000
TOTAL: 1 objects, 211084829 bytes (201.31 MiB)
168.31 MiB 2022-03-25T15:38:16Z gs:///01-100000
TOTAL: 1 objects, 176486219 bytes (168.31 MiB)
178.05 MiB 2022-03-25T15:38:15Z gs:///01-110000
TOTAL: 1 objects, 186697415 bytes (178.05 MiB)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the shown fs.put call and paired local and GCS paths, then inspect the put entry point and its handling of multiple paths. Done means each local path is written to the corresponding remote path, with a regression test covering the mapping.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.