Py_UNICODE_TOUPPER() and friends do not return the simple case mappings

Bug report

Py_UNICODE_TOUPPER(), Py_UNICODE_TOLOWER() and Py_UNICODE_TOTITLE() return the first character of the full case mapping when it is longer than one character.

UnicodeData.txt leaves the simple uppercase of 'ß' (LATIN SMALL LETTER SHARP S) empty, which means the character is its own simple uppercase; the full uppercase "SS" is in SpecialCasing.txt. Py_UNICODE_TOUPPER() returns 'S'. The same for 'ǰ' -> 'J', 'fi' -> 'F' and 'ΐ' -> 'Ι'.

For 27 Greek letters with ypogegrammeni the UCD does define a simple uppercase, and it is not the first character of the full one:

1F80;GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI;Ll;0;L;1F00 0345;;;;N;;;1F88;;1F88
1F80; 1F80; 1F88; 1F08 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI

The simple uppercase of 'ᾀ' is 'ᾈ' (U+1F88), the full one is Ἀ + Ι, and the macro returns Ἀ (U+1F08). Tools/unicode/makeunicodedata.py stores only the full mappings for a character which has a SpecialCasing.txt entry, so the simple ones are not in the table at all.

102 characters are affected for the uppercase, 48 for the titlecase, and one (LATIN CAPITAL LETTER I WITH DOT ABOVE, whose simple lowercase is 'i') for the lowercase. All of them are Unicode 1.1, and no character added since has joined them.

This is a side effect of b2bf01d824e (bpo-12736, 3.3), which added the full mappings: before it the fields held the simple mappings.

_sre is the only user in CPython, and it compensates: _sre.unicode_iscased() reports whether the simple lowercase or uppercase differs from the character, which is right for 'ß' only because the returned 'S' differs from it. Fixing the mappings requires comparing the full mappings there instead.

Linked PRs

• gh-156519

現在 SpecialCasing.txt のエントリに対する完全なマッピングを保存している Tools/unicode/makeunicodedata.py から始め、次に _sre unicode_iscased() の補正を調べます。影響を受ける文字について単純な大文字・小文字マッピングが保持され、_sre が完全なマッピングを使用して大文字・小文字を引き続き正しく判定できれば完了です。