aws / aws/aws-cli

encoding problem of aws cloudformation package

Open
#7,142 2 comments 1 reaction 0 assignees View on GitHub
bug cloudformation package-deploy customization locale p3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the bug

Even if we specify AWS_CLI_FILE_ENCODING=utf-8,
aws cloudformation package with --output-template-file create a non utf-8 output.
It depends on locale because python build-in open function is used.
Input template file is utf-8, but output template file is non utf-8.
This sometime leads an encode error.

When we use nested stacks,
aws cli create a temporary file without an encoding before uploading.
There is the same encoding problem.

### Expected Behavior

aws cloudformation package command should create a file with an encoding of AWS_CLI_FILE_ENCODING.

### Current Behavior

Encode Error occur.
`'cp932' codec can't encode character '\U0002000b' in position 186: illegal multibyte sequence`
In this case, I used aws cli on Winodws.

### Reproduction Steps

- prepare cloudformation nested stack template files with utf-8 (include non-ascii characters, such as surrogate pair U+2000B="D840 DC0B")
- export AWS_CLI_FILE_ENCODING=utf-8
- aws cloudformation package with --output-template-file option

### Possible Solution

I suggest that use compat_open() or getpreferredencoding() of aws-cli/compat.py for an output file encoding.

built-in open function is used in
- [https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/package.py#L180](https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/package.py#L180)
- [https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/artifact_exporter.py#L206](https://github.com/aws/aws-cli/blob/2.7.18/awscli/customizations/cloudformation/artifact_exporter.py#L206)

Work arounds:
- PYTHONUTF8=1 (utf-8 mode, python 3.7+) (available on both Widows and Linux)
- LC_CTYPE=UTF-8 (Linux only)

### Additional Information/Context

Conjunction use of "AWS_CLI_FILE_ENCODING" and "LC_CTYPE" is not a better solution.
Because fewer locales are available for "LC_CTYPE" on CodeBuild by default, so we need install a language pack additionally.
Locale is difficult to change on Windows.
Python can handle encodings without an additional library.

### CLI version used

2.7.18

### Environment details (OS name and version, etc.)

Windows 10

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.