GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil doesn't gracefully handle non-UTF8 filenames
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I use `gsutil` to rsync files from a public ftp drive to a GCS bucket. Since the ftp allows public access, our users sometimes upload files with non-UTF8 filename encodings (i.e. Windows Code Pages for Japanese, Chinese, etc). When this happens, I get the following error when gsutil encounters the non-UTF8 filename during `rsync`:
```
Caught non-retryable exception while listing file:///mnt/data/proftpd/data/ftps/projectdrop/: 'utf-8' codec can't encode characters in position 50-54: surrogates not allowed
```
The problematic filename appears as `'PS4'$'\222\267\216\236\212''Ԓ'$'\342\216''~.zip.001'` on Linux, and when viewed in `mc` the chars appear as gibberish:

A Google search for the error message *surrogates not allowed* turned up [this](https://stackoverflow.com/questions/27366479/python-3-os-walk-file-paths-unicodeencodeerror-utf-8-codec-cant-encode-s) SO post. Apparently this is a Python 3 issue which affects `gsutil` as it uses Python 3:
> Python 3 tries to turn everything into Unicode strings. In doing so the developers came up with a scheme to translate byte strings to Unicode strings and back without loss, and without knowing the original encoding. They used partial surrogates to encode the 'bad' bytes, but the normal UTF8 encoder can't handle them when printing to the terminal.
I was able to work around this issue by manually renaming the non-UTF8 filename to something POSIX-friendly and re-running `gsutil`. It would be nice, however, if `gsutil` was able to handle this edge case, perhaps by selecting some arbitrary utf8 string to replace the bad filename with incrementing, i.e. `bad-encoding-{001..n}`. Thanks!
Contributor guide
Research direction
Start at the gsutil rsync path that lists local filesystem entries under Python 3, then reproduce the reported non-UTF8 filename case. Trace how surrogate-containing filenames are encoded or reported; done means rsync handles the entry without raising the reported UnicodeEncodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100