GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Improve unicode handling

Open
#139 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Right now, there are sporadic utf-8 encode and decodes throughout the code. We should do a few things:
- Centralize unicode conversions into util. The best practice is to define functions like `smart_bytes` and `smart_text` from [django's utils](https://github.com/django/django/blob/master/django/utils/encoding.py). These should be used instead of manually calling .decode or .encode. This will also help when supporting Python 3.
- Change all output commands (print, sys.stdout.write, sys.stderr.write, etc) to use smart_bytes or equivalent. Right now, we sometimes print utf-8 strings directly, which is not correct. This almost always works because the default terminal encoding is usually utf-8, so it gets decoded properly. However, when you run with a piped process that doesn't use utf-8 streams (for example subprocess.call) things can break. All output should look like `sys.stdout.write(smart_bytes(something))`

Part of this was started with [8357045](https://codereview.appspot.com/8357045/) and [8274045](https://codereview.appspot.com/8274045/)

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing .encode and .decode calls, output commands, and the util module. Compare the proposed smart_bytes and smart_text functions with Django's encoding utilities and review the linked prior work. Done means conversions are centralized and command output consistently handles non-UTF-8 streams, including the stated Python 3 compatibility goal.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.