Py_UNICODE_TOUPPER() and friends do not return the simple case mappings

Bug report

Py_UNICODE_TOUPPER(), Py_UNICODE_TOLOWER() and Py_UNICODE_TOTITLE() return the first character of the full case mapping when it is longer than one character.

UnicodeData.txt leaves the simple uppercase of 'ß' (LATIN SMALL LETTER SHARP S) empty, which means the character is its own simple uppercase; the full uppercase "SS" is in SpecialCasing.txt. Py_UNICODE_TOUPPER() returns 'S'. The same for 'ǰ' -> 'J', 'fi' -> 'F' and 'ΐ' -> 'Ι'.

For 27 Greek letters with ypogegrammeni the UCD does define a simple uppercase, and it is not the first character of the full one:

1F80;GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI;Ll;0;L;1F00 0345;;;;N;;;1F88;;1F88
1F80; 1F80; 1F88; 1F08 0399; # GREEK SMALL LETTER ALPHA WITH PSILI AND YPOGEGRAMMENI

The simple uppercase of 'ᾀ' is 'ᾈ' (U+1F88), the full one is Ἀ + Ι, and the macro returns Ἀ (U+1F08). Tools/unicode/makeunicodedata.py stores only the full mappings for a character which has a SpecialCasing.txt entry, so the simple ones are not in the table at all.

102 characters are affected for the uppercase, 48 for the titlecase, and one (LATIN CAPITAL LETTER I WITH DOT ABOVE, whose simple lowercase is 'i') for the lowercase. All of them are Unicode 1.1, and no character added since has joined them.

This is a side effect of b2bf01d824e (bpo-12736, 3.3), which added the full mappings: before it the fields held the simple mappings.

_sre is the only user in CPython, and it compensates: _sre.unicode_iscased() reports whether the simple lowercase or uppercase differs from the character, which is right for 'ß' only because the returned 'S' differs from it. Fixing the mappings requires comparing the full mappings there instead.

Linked PRs

• gh-156519

Start with Tools/unicode/makeunicodedata.py, which currently stores full mappings for SpecialCasing.txt entries, then inspect the _sre unicode_iscased() compensation. Done means simple case mappings are preserved for the affected characters and _sre still determines casing correctly using the full mappings.