aws / aws/aws-cli

Not encoding Unicode characters in json output can lead to `UnicodeEncodeError`.

Open
#6,942 4 comments 1 reaction 0 assignees View on GitHub
dynamodb feature-request p2 unicode
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

I queried a dynamoDB and asked for json output. The result contained Unicode chars that cannot be encoded in cp1252, the default character encoding for console applications on Windows. This raised a `UnicodeEncodeError` in AWS CLI halfway through the output.

### Expected Behavior

AWS CLI should output valid json and not throw an exception, regardless of what the encoding is for stdout.

### Current Behavior

AWS CLI explicitly tells the json encoder to _not_ ensure all output characters are in ASCII range:
https://github.com/aws/aws-cli/blob/develop/awscli/formatter.py#L95
```
json.dump(response, stream, indent=4, default=json_encoder,
ensure_ascii=False)
```
The json encoder will consequently not encode Unicode characters and try to write them to stdout:
https://github.com/python/cpython/blob/main/Lib/json/__init__.py#L179
```
for chunk in iterable:
fp.write(chunk)
```
At this point, the Unicode characters will automatically be encoded to whatever encoding stdout is using. On Windows, this is commonly [cp1252](https://en.wikipedia.org/wiki/Windows-1252), which does not cover the entire Unicode range. If the output contains any Unicode characters that cannot be encoded, it will raise a `UnicodeEncodeError`.

### Reproduction Steps

1. Create a dynamoDB table with a field that contains a Unicode character that cannot be encoded in cp1252, such as "⚠".
2. Run this command `aws dynamodb scan --table-name --output json`

### Possible Solution

Allow json output to be ASCII only. Either by default, or through a command-line flag.

### Additional Information/Context

_No response_

### CLI version used

aws-cli/2.3.0 Python/3.8.8 Windows/10 exe/AMD64 prompt/off

### Environment details (OS name and version, etc.)

Windows 10 20H2 (OS Build 19042.1645), Python 3.10.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.