sillsdev / sillsdev/icu-dotnet
Normalizer.Normalize silently does NFKD for UNORM_FCD and UNORM_NONE
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 83
- Forks
- 33
- Avg merge
- 3h 21m
- Merged PRs (30d)
- 1
Description
Normalizer.GetNormalizer picks the instance name and mode from two ternaries:
return Normalizer2.GetInstance(null,
mode == UNORM_NFC || mode == UNORM_NFD ? "nfc" : "nfkc",
mode == UNORM_NFC || mode == UNORM_NFKC ? COMPOSE : DECOMPOSE);
UNORM_FCD and UNORM_NONE both fall through to "nfkc" + DECOMPOSE, i.e. NFKD. So
Normalizer.Normalize(s, UNORM_FCD) returns NFKD and Normalizer.IsNormalized(s, UNORM_FCD)
answers the NFKD question, with no error.
Normalizer2.Mode already has FCD, so the FCD case can map to "nfc" + Mode.FCD.
UNORM_NONE has no Normalizer2 equivalent and should probably throw rather than quietly
normalize.
Found while measuring whether NFC output is FCD-conformant for
https://github.com/sillsdev/icu-dotnet/issues/130 — it isn't always, but the check reported
through UNORM_FCD was answering a different question entirely.
source/icu.net/Normalization/Normalizer.cs:35-43
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in source/icu.net/Normalization/Normalizer.cs:35-43 and inspect Normalizer.GetNormalizer’s mode mapping. Verify Normalize and IsNormalized for UNORM_FCD, then ensure FCD uses the existing Normalizer2 FCD mode and UNORM_NONE no longer silently selects NFKD; done means both modes have the intended behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100