GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil -m rsync fails between s3 and gs when Content-Type is missing in S3
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Sorry if this is a dupe, or a boto issue rather than a gsutil issue. I couldn't find a solution, hence this issue.
I am trying to perform a daily rsync from s3 to gs of hundreds of thousands of files. Some of the original files in s3 appear to have no Content-Type, for unknown reasons. I expect this could happen at any time to future files in s3 too.
My rsync fails like so:
```
Copying s3://a_bucket_name/a_dir/filename.xls [Content-Type=; charset=binary]...
need more than 1 value to unpackat...: 0 B/266 KB
```
on multiple files. These do have blank Content-Type headers on s3.
`use_magicfile = True` doesn't appear to help to use local `file` to make a guess at Content-Type.
Similarly, trying to not care about Content-Type globally with `-h 'Content-Type:'` fails with:
`CommandException: option -h not recognized for "rsync" command.`
Is there a way I can avoid exceptions, or provide a (even incorrect) Content-Type by default when one is unknown?
Contributor guide
Assessment
This issue has not been assessed yet.