aws / aws/aws-cli

Provide an option to perfom unicode normalization on local file names

Open
#1,639 14 comments 8 reactions 0 assignees View on GitHub
feature-request p3 s3 unicode
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

## Summary

`aws s3 sync` doesn't play well with HFS+ unicode normalization on OS X. I suggest to add an option to normalize file names read locally in normal form C before doing anything with them.
## Reproduction steps
1. Create a file on S3 containing an accented character. For reasons that will become apparent later, do this on a Linux system.

```
(linux) % echo test > test/café.txt
(linux) % aws s3 sync test s3:///test
```
2. Synchronize that file on a Mac.

```
(OS X) % aws s3 sync s3:///test test
download: s3:///test/café.txt to test/café.txt
```
3. Synchronize it back to S3.

```
(OS X) % aws s3 sync s3:///test test
upload: test/café.txt to s3:///test/café.txt
```
- Expected result: no upload because the file is identical locally and on S3: I was just sync'd!
- Actual result: the file is uploaded again.

At this point the file shows up twice in S3!

screen shot 2015-11-14 at 22 45 38

## Why this happens

Unicode defines two normal forms — NFC and NFD — for some characters, typically accented characters which are common in Western European languages and even occur in English.

The documentation of [unicodedata.normalize](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize), the Python function that converts between the two forms, has a good explanation.

A quick illustration:

```
>>> "café".encode('utf-8')
b'caf\xc3\xa9'
>>> unicodedata.normalize('NFC', "café").encode('utf-8')
b'caf\xc3\xa9'
>>> unicodedata.normalize('NFD', "café").encode('utf-8')
b'cafe\xcc\x81'
```

The default filesystem of OS X, [HFS+](https://en.wikipedia.org/wiki/HFS_Plus), enforces something that resembles NFD. (Let's say I haven't encountered the difference yet.)

Pretty much everything else, including typing on a keyboard on Linux or OS X, uses NFC. I'm not sure about Windows.

Of course this is entirely HFS+'s fault, but since OS X is a popular system among your target audience, I hope you may have some interest in providing a solution to this problem.

## What you can do about it

I think a `--normalize-unicode` option (possibly with a better name) for `aws s3 sync` would be useful. It would normalize file names read from the local filesystem with `unicodedata.normalize('NFKC', filepath)`.

Its primary purpose would be to interact with S3 on OS X and have file names in NFC form on S3, which is what the rest of the world expects and will cause the least amount of problems.

I don't know `aws cli` well enough to tell which other parts could use this option. I just encountered the problem when trying to replace "`rsync` to file server" with "`aws s3 sync` to S3".

FWIW `rsync` provides a solution to this problem with the `--iconv` option. A common idiom is `--iconv=UTF8-MAC,UTF8` when rsync'ing from OS X to Linux and `--iconv=UTF8,UTF8-MAC` when rsync'ing from Linux to OS X. `UTF8-MAC` is how `rsync` calls the encoding of file names on HFS+.

However this isn't a good API to tackle the specific problem I'm raising here. This API is about the encoding of file names. The bug is related to Unicode normalization. These are different concepts. `UTF8-MAC` mixes them.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.