GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil doesn't gracefully handle non-UTF8 filenames

Open
#1,625 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I use `gsutil` to rsync files from a public ftp drive to a GCS bucket. Since the ftp allows public access, our users sometimes upload files with non-UTF8 filename encodings (i.e. Windows Code Pages for Japanese, Chinese, etc). When this happens, I get the following error when gsutil encounters the non-UTF8 filename during `rsync`:

```
Caught non-retryable exception while listing file:///mnt/data/proftpd/data/ftps/projectdrop/: 'utf-8' codec can't encode characters in position 50-54: surrogates not allowed
```

The problematic filename appears as `'PS4'$'\222\267\216\236\212''Ԓ'$'\342\216''~.zip.001'` on Linux, and when viewed in `mc` the chars appear as gibberish:
![non-utf8-filename](https://user-images.githubusercontent.com/1127733/203465227-bcc13c14-3502-48a8-b1b2-536b36830dbf.png)

A Google search for the error message *surrogates not allowed* turned up [this](https://stackoverflow.com/questions/27366479/python-3-os-walk-file-paths-unicodeencodeerror-utf-8-codec-cant-encode-s) SO post. Apparently this is a Python 3 issue which affects `gsutil` as it uses Python 3:

> Python 3 tries to turn everything into Unicode strings. In doing so the developers came up with a scheme to translate byte strings to Unicode strings and back without loss, and without knowing the original encoding. They used partial surrogates to encode the 'bad' bytes, but the normal UTF8 encoder can't handle them when printing to the terminal.

I was able to work around this issue by manually renaming the non-UTF8 filename to something POSIX-friendly and re-running `gsutil`. It would be nice, however, if `gsutil` was able to handle this edge case, perhaps by selecting some arbitrary utf8 string to replace the bad filename with incrementing, i.e. `bad-encoding-{001..n}`. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start at the gsutil rsync path that lists local filesystem entries under Python 3, then reproduce the reported non-UTF8 filename case. Trace how surrogate-containing filenames are encoded or reported; done means rsync handles the entry without raising the reported UnicodeEncodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.