dotnet / dotnet/runtime

ToUpperOrdinal canonicalization disagrees with OrdinalIgnoreCase for a supplementary Deseret pair under NLS

Open
#133,950 1 comment 0 reactions 0 assignees View on GitHub
area-System.Globalization untriaged
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

`string.ToUpperOrdinal()` documents a canonicalization relationship with
`StringComparison.OrdinalIgnoreCase`: two strings are ordinal-ignore-case equal
if and only if their ordinal-uppercase forms are ordinally equal.

That contract does not hold for a Deseret supplementary-character case when
.NET uses Windows NLS:

- `U+10428 DESERET SMALL LETTER LONG I`
- `U+10400 DESERET CAPITAL LETTER LONG I`

Under NLS, `StringComparison.OrdinalIgnoreCase` reports these strings as
different, while `ToUpperOrdinal()` maps both strings to U+10400.

This blocks consumers such as ASP.NET Core Output Caching from using
`ToUpperOrdinal()` as a collision-free canonical representation for keys whose
equivalence must match routing's ordinal-ignore-case semantics.

### Reproduction

```csharp
using System.Runtime.InteropServices;
using System.Text;

var cases = new (string Name, string Left, string Right)[]
{
("ASCII case control", "s", "S"),
("long-s BMP control", "s", "\u017F"),
("Kelvin BMP control", "k", "\u212A"),
("sigma BMP control", "\u03C2", "\u03C3"),
("Deseret supplementary regression", "\U00010428", "\U00010400"),
};

Console.WriteLine($"Framework: {RuntimeInformation.FrameworkDescription}");
Console.WriteLine($"OS: {RuntimeInformation.OSDescription}");
Console.WriteLine($"DOTNET_SYSTEM_GLOBALIZATION_USENLS: {Environment.GetEnvironmentVariable("DOTNET_SYSTEM_GLOBALIZATION_USENLS") ?? ""}");
Console.WriteLine($"DOTNET_SYSTEM_GLOBALIZATION_INVARIANT: {Environment.GetEnvironmentVariable("DOTNET_SYSTEM_GLOBALIZATION_INVARIANT") ?? ""}");

var failures = 0;
foreach (var (name, left, right) in cases)
{
var leftCanonical = left.ToUpperOrdinal();
var rightCanonical = right.ToUpperOrdinal();
var originalsEqual = left.Equals(right, StringComparison.OrdinalIgnoreCase);
var canonicalFormsEqual = leftCanonical.Equals(rightCanonical, StringComparison.Ordinal);
var passed = originalsEqual == canonicalFormsEqual;

Console.WriteLine($"{(passed ? "PASS" : "FAIL")}: {name}");
Console.WriteLine($" originals OIC-equal: {originalsEqual}");
Console.WriteLine($" canonical ordinal-equal: {canonicalFormsEqual}");
Console.WriteLine($" canonical values: {Format(leftCanonical)} / {Format(rightCanonical)}");
failures += passed ? 0 : 1;
}

return failures == 0 ? 0 : 1;

static string Format(string value) =>
string.Join(" ", value.EnumerateRunes().Select(static rune => $"U+{rune.Value:X4}"));
```

Run on Windows with NLS explicitly enabled:

```powershell
$env:DOTNET_SYSTEM_GLOBALIZATION_USENLS = "1"
dotnet run
```

### Actual result

On .NET 11.0.0-rc.1.26420.103, Windows 10.0.26200 x64:

```text
PASS: ASCII case control
PASS: long-s BMP control
PASS: Kelvin BMP control
PASS: sigma BMP control
FAIL: Deseret supplementary regression
originals OIC-equal: False
canonical ordinal-equal: True
canonical values: U+10400 / U+10400
```

The process exits with code 1. The same complete test passes under default ICU
and invariant globalization, where the Deseret originals are
ordinal-ignore-case equal and both canonicalize to U+10400.

### Expected result

For every pair of strings and every supported globalization mode:

```csharp
left.Equals(right, StringComparison.OrdinalIgnoreCase)
==
left.ToUpperOrdinal().Equals(right.ToUpperOrdinal(), StringComparison.Ordinal)
```

Runtime regression coverage should include supplementary Unicode scalars in
addition to the existing exhaustive BMP checks, and should execute under
Windows NLS.

### Additional context

The public API was approved in #90999 and implemented in #130140 specifically
to provide this iff relationship. Current tests exhaust the BMP and separately
verify Rune casing for Deseret, but do not test the canonicalization iff
contract for supplementary pairs under NLS.

Under NLS, ordinal-ignore-case comparison delegates to Windows
`CompareStringOrdinal`, while ordinal casing maps the supplementary Deseret
Rune to its capital form. These surfaces need to agree before
`ToUpperOrdinal()` can safely serve as an ordinal-ignore-case key
canonicalizer.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied Deseret pair on Windows with DOTNET_SYSTEM_GLOBALIZATION_USENLS set to 1, then read the existing exhaustive BMP and Deseret Rune-casing tests. Add regression coverage for the canonicalization iff relationship with supplementary scalars under NLS, while preserving the existing controls. Done means the test passes under Windows NLS as well as the other globalization modes described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
operating-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.