GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Downloading (content type) compressed files breaks both cp -n and rsync
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Typical common, trivial use case:
Attempt to cp -r -n or rsync a GooglePlay developer reports bucket to local disk. (This is a bucket created and maintained by GooglePlay, so there is no option to change content storage/encoding.)
Any compressed files will be re-copied every time, despite cp -n or rsync use.
In the cp -n case, I am guessing it is due to this line in copy_helper.py:
```
# The local file may be a partial. Check the file sizes.
if src_obj_size == os.path.getsize(dst_url.object_name):
raise ItemExistsError()
```
which notices the (uncompressed) destination is not the same size as the (compressed) source, and so assumes a partial prior download.
Possible solution: Provide a flag which automatically adds a .gz extention to the target name and does Not decompress target file locally. This may cleanly extend existing no-clobber logic to compressed files?
[Note also that current logic technically has a (rare) bug: If a partial download happens to decompress to same size as full compressed file, it will not retry due to false no-clobber.]
Contributor guide
Assessment
This issue has not been assessed yet.