sillsdev / sillsdev/icu-dotnet

Normalizer.Normalize silently does NFKD for UNORM_FCD and UNORM_NONE

Open Beginner friendly
#232 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
83
Forks
33
Avg merge
3h 21m
Merged PRs (30d)
1

Description

Normalizer.GetNormalizer picks the instance name and mode from two ternaries:

return Normalizer2.GetInstance(null,
    mode == UNORM_NFC || mode == UNORM_NFD ? "nfc" : "nfkc",
    mode == UNORM_NFC || mode == UNORM_NFKC ? COMPOSE : DECOMPOSE);

UNORM_FCD and UNORM_NONE both fall through to "nfkc" + DECOMPOSE, i.e. NFKD. So
Normalizer.Normalize(s, UNORM_FCD) returns NFKD and Normalizer.IsNormalized(s, UNORM_FCD)
answers the NFKD question, with no error.

Normalizer2.Mode already has FCD, so the FCD case can map to "nfc" + Mode.FCD.
UNORM_NONE has no Normalizer2 equivalent and should probably throw rather than quietly
normalize.

Found while measuring whether NFC output is FCD-conformant for
https://github.com/sillsdev/icu-dotnet/issues/130 — it isn't always, but the check reported
through UNORM_FCD was answering a different question entirely.

source/icu.net/Normalization/Normalizer.cs:35-43

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in source/icu.net/Normalization/Normalizer.cs:35-43 and inspect Normalizer.GetNormalizer’s mode mapping. Verify Normalize and IsNormalized for UNORM_FCD, then ensure FCD uses the existing Normalizer2 FCD mode and UNORM_NONE no longer silently selects NFKD; done means both modes have the intended behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.