encoding problem of aws cloudformation package
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
Even if we specify AWS_CLI_FILE_ENCODING=utf-8,
aws cloudformation package with --output-template-file create a non utf-8 output.
It depends on locale because python build-in open function is used.
Input template file is utf-8, but output template file is non utf-8.
This sometime leads an encode error.
When we use nested stacks,
aws cli create a temporary file without an encoding before uploading.
There is the same encoding problem.
### Expected Behavior
aws cloudformation package command should create a file with an encoding of AWS_CLI_FILE_ENCODING.
### Current Behavior
Encode Error occur.
`'cp932' codec can't encode character '\U0002000b' in position 186: illegal multibyte sequence`
In this case, I used aws cli on Winodws.
### Reproduction Steps
- prepare cloudformation nested stack template files with utf-8 (include non-ascii characters, such as surrogate pair U+2000B="D840 DC0B")
- export AWS_CLI_FILE_ENCODING=utf-8
- aws cloudformation package with --output-template-file option
### Possible Solution
I suggest that use compat_open() or getpreferredencoding() of aws-cli/compat.py for an output file encoding.
built-in open function is used in
- [https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/package.py#L180](https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/package.py#L180)
- [https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/artifact_exporter.py#L206](https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/artifact_exporter.py#L206)
Work arounds:
- PYTHONUTF8=1 (utf-8 mode, python 3.7+) (available on both Widows and Linux)
- LC_CTYPE=UTF-8 (Linux only)
### Additional Information/Context
Conjunction use of "AWS_CLI_FILE_ENCODING" and "LC_CTYPE" is not a better solution.
Because fewer locales are available for "LC_CTYPE" on CodeBuild by default, so we need install a language pack additionally.
Locale is difficult to change on Windows.
Python can handle encodings without an additional library.
### CLI version used
2.7.18
### Environment details (OS name and version, etc.)
Windows 10
Contributor guide
Assessment
This issue has not been assessed yet.