Provide an option to perfom unicode normalization on local file names
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
## Summary
`aws s3 sync` doesn't play well with HFS+ unicode normalization on OS X. I suggest to add an option to normalize file names read locally in normal form C before doing anything with them.
## Reproduction steps
1. Create a file on S3 containing an accented character. For reasons that will become apparent later, do this on a Linux system.
```
(linux) % echo test > test/café.txt
(linux) % aws s3 sync test s3:///test
```
2. Synchronize that file on a Mac.
```
(OS X) % aws s3 sync s3:///test test
download: s3:///test/café.txt to test/café.txt
```
3. Synchronize it back to S3.
```
(OS X) % aws s3 sync s3:///test test
upload: test/café.txt to s3:///test/café.txt
```
- Expected result: no upload because the file is identical locally and on S3: I was just sync'd!
- Actual result: the file is uploaded again.
At this point the file shows up twice in S3!

## Why this happens
Unicode defines two normal forms — NFC and NFD — for some characters, typically accented characters which are common in Western European languages and even occur in English.
The documentation of [unicodedata.normalize](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize), the Python function that converts between the two forms, has a good explanation.
A quick illustration:
```
>>> "café".encode('utf-8')
b'caf\xc3\xa9'
>>> unicodedata.normalize('NFC', "café").encode('utf-8')
b'caf\xc3\xa9'
>>> unicodedata.normalize('NFD', "café").encode('utf-8')
b'cafe\xcc\x81'
```
The default filesystem of OS X, [HFS+](https://en.wikipedia.org/wiki/HFS_Plus), enforces something that resembles NFD. (Let's say I haven't encountered the difference yet.)
Pretty much everything else, including typing on a keyboard on Linux or OS X, uses NFC. I'm not sure about Windows.
Of course this is entirely HFS+'s fault, but since OS X is a popular system among your target audience, I hope you may have some interest in providing a solution to this problem.
## What you can do about it
I think a `--normalize-unicode` option (possibly with a better name) for `aws s3 sync` would be useful. It would normalize file names read from the local filesystem with `unicodedata.normalize('NFKC', filepath)`.
Its primary purpose would be to interact with S3 on OS X and have file names in NFC form on S3, which is what the rest of the world expects and will cause the least amount of problems.
I don't know `aws cli` well enough to tell which other parts could use this option. I just encountered the problem when trying to replace "`rsync` to file server" with "`aws s3 sync` to S3".
FWIW `rsync` provides a solution to this problem with the `--iconv` option. A common idiom is `--iconv=UTF8-MAC,UTF8` when rsync'ing from OS X to Linux and `--iconv=UTF8,UTF8-MAC` when rsync'ing from Linux to OS X. `UTF8-MAC` is how `rsync` calls the encoding of file names on HFS+.
However this isn't a good API to tackle the specific problem I'm raising here. This API is about the encoding of file names. The bug is related to Unicode normalization. These are different concepts. `UTF8-MAC` mixes them.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.