GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Improve unicode handling
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Right now, there are sporadic utf-8 encode and decodes throughout the code. We should do a few things:
- Centralize unicode conversions into util. The best practice is to define functions like `smart_bytes` and `smart_text` from [django's utils](https://github.com/django/django/blob/master/django/utils/encoding.py). These should be used instead of manually calling .decode or .encode. This will also help when supporting Python 3.
- Change all output commands (print, sys.stdout.write, sys.stderr.write, etc) to use smart_bytes or equivalent. Right now, we sometimes print utf-8 strings directly, which is not correct. This almost always works because the default terminal encoding is usually utf-8, so it gets decoded properly. However, when you run with a piped process that doesn't use utf-8 streams (for example subprocess.call) things can break. All output should look like `sys.stdout.write(smart_bytes(something))`
Part of this was started with [8357045](https://codereview.appspot.com/8357045/) and [8274045](https://codereview.appspot.com/8274045/)
Contributor guide
Research direction
Start by locating the existing .encode and .decode calls, output commands, and the util module. Compare the proposed smart_bytes and smart_text functions with Django's encoding utilities and review the linked prior work. Done means conversions are centralized and command output consistently handles non-UTF-8 streams, including the stated Python 3 compatibility goal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100