Not encoding Unicode characters in json output can lead to `UnicodeEncodeError`.
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
I queried a dynamoDB and asked for json output. The result contained Unicode chars that cannot be encoded in cp1252, the default character encoding for console applications on Windows. This raised a `UnicodeEncodeError` in AWS CLI halfway through the output.
### Expected Behavior
AWS CLI should output valid json and not throw an exception, regardless of what the encoding is for stdout.
### Current Behavior
AWS CLI explicitly tells the json encoder to _not_ ensure all output characters are in ASCII range:
https://github.com/aws/aws-cli/blob/develop/awscli/formatter.py#L95
```
json.dump(response, stream, indent=4, default=json_encoder,
ensure_ascii=False)
```
The json encoder will consequently not encode Unicode characters and try to write them to stdout:
https://github.com/python/cpython/blob/main/Lib/json/__init__.py#L179
```
for chunk in iterable:
fp.write(chunk)
```
At this point, the Unicode characters will automatically be encoded to whatever encoding stdout is using. On Windows, this is commonly [cp1252](https://en.wikipedia.org/wiki/Windows-1252), which does not cover the entire Unicode range. If the output contains any Unicode characters that cannot be encoded, it will raise a `UnicodeEncodeError`.
### Reproduction Steps
1. Create a dynamoDB table with a field that contains a Unicode character that cannot be encoded in cp1252, such as "⚠".
2. Run this command `aws dynamodb scan --table-name --output json`
### Possible Solution
Allow json output to be ASCII only. Either by default, or through a command-line flag.
### Additional Information/Context
_No response_
### CLI version used
aws-cli/2.3.0 Python/3.8.8 Windows/10 exe/AMD64 prompt/off
### Environment details (OS name and version, etc.)
Windows 10 20H2 (OS Build 19042.1645), Python 3.10.0
Contributor guide
Assessment
This issue has not been assessed yet.