GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Downloading (content type) compressed files breaks both cp -n and rsync

Open
#379 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Typical common, trivial use case:

Attempt to cp -r -n or rsync a GooglePlay developer reports bucket to local disk. (This is a bucket created and maintained by GooglePlay, so there is no option to change content storage/encoding.)

Any compressed files will be re-copied every time, despite cp -n or rsync use.

In the cp -n case, I am guessing it is due to this line in copy_helper.py:

```
# The local file may be a partial. Check the file sizes.
if src_obj_size == os.path.getsize(dst_url.object_name):
raise ItemExistsError()
```

which notices the (uncompressed) destination is not the same size as the (compressed) source, and so assumes a partial prior download.

Possible solution: Provide a flag which automatically adds a .gz extention to the target name and does Not decompress target file locally. This may cleanly extend existing no-clobber logic to compressed files?

[Note also that current logic technically has a (rare) bug: If a partial download happens to decompress to same size as full compressed file, it will not retry due to false no-clobber.]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.