Unicode Normalization of Identifiers

Related to #2287

All identifiers must be normalized to the NFC form.
Normalization is used to treat equally strings which consists of different codepoints but practiaclly the same.
It is useful when a program is linked to other Rust libraries,

NFC Normalization takes two steps:

1. Some kinds of characters are decomposed to one or multiple chatacters. (This is called NFD form)
2. Then, some kinds of chatacters are composed to a single character. (This is called NFC form)

For example,
https://unicode.org/reports/tr15/images/UAX15-NormFig4.jpg

Implementation

Unicode data file parser

• written in python
  • https://github.com/Rust-GCC/gccrs/pull/2425
• add type annotation for mypy
  • https://github.com/Rust-GCC/gccrs/pull/2529

Decomposition

• recursive decomposition with Decompostion_Mapping
  • https://github.com/Rust-GCC/gccrs/pull/2425
• algorithmic decomposition of Hangul
• sort characters with Canonical_Combining_Class
  • https://github.com/Rust-GCC/gccrs/pull/2425
• algorithmic decomposition of Hangul
  • https://github.com/Rust-GCC/gccrs/pull/2467

Composition

• recompose a starter with its following chartacters if composable
  • https://github.com/Rust-GCC/gccrs/pull/2425
• do not recompose characters with the Composition_Exclusion property
  • https://github.com/Rust-GCC/gccrs/pull/2425
• algorithmic composition of Hangul
  • https://github.com/Rust-GCC/gccrs/pull/2467

Optimization

• quick check
• use trie for composition mapping table
  • but the current implementation is not so slow

Tests

many testcases are needed!

Ref

• https://doc.rust-lang.org/reference/identifiers.html#normalization
• https://unicode.org/reports/tr15/
• Reference implemetation: https://www.w3.org/International/charlint/

Review the Unicode data file parser and the decomposition and composition work described in the issue, including the linked implementation pull requests. Then assess the remaining optimization items, especially the quick check and composition mapping trie, and expand the unspecified test coverage. Done means identifiers are normalized to NFC and the needed normalization tests pass.