NVIDIA / NVIDIA/cudf

[BUG] strings::to_upper / to_lower silently skip case-folding for supplementary (non-BMP) Unicode code points

Open
#24,205 1 comment 0 reactions 0 assignees View on GitHub
? - Needs Triage bug wontfix
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

`cudf::strings::to_upper`, `to_lower`, and `swapcase` produce silently wrong results for any string containing a cased Unicode character above U+FFFF (supplementary plane). The character is passed through **unchanged** instead of being case-folded. No error or warning is raised.

BMP characters (ASCII, Latin, accented characters like `Ö`) are case-folded correctly. Only code points above U+FFFF are affected.

**Steps/Code to reproduce bug**

Minimal Python repro:

```python
import cudf

# U+1042D DESERET SMALL LETTER EW → U+10405 DESERET CAPITAL LETTER EW
s = cudf.Series(["From\U0001042DTo"]) # "From𐐭To"
print(s.str.upper().to_pandas()[0])
# Expected: FROM𐐅TO
# Actual: FROM𐐭TO ← supplementary character is unchanged
```

The equivalent pandas call returns the correct result:

```python
import pandas as pd
pd.Series(["From\U0001042DTo"]).str.upper()[0]
# 'FROM𐐅TO' ← correct
```

Same failure for `str.lower()` and `str.swapcase()` on any cased code point above U+FFFF:

| Input | Operation | Expected | Actual |
|---|---|---|---|
| `𐐭` U+1042D Deseret Small Letter EW | `to_upper` | `𐐅` U+10405 | `𐐭` unchanged |
| `𐐅` U+10405 Deseret Capital Letter EW | `to_lower` | `𐐭` U+1042D | `𐐅` unchanged |
| `𞤢` U+1E922 Adlam Small Letter Alef | `to_upper` | `𞤀` U+1E900 | `𞤢` unchanged |

**Expected behavior**

Supplementary-plane cased characters should be folded to their Unicode case equivalent, consistent with how BMP characters are handled.

**Root cause**

In `cpp/src/strings/case.cu`, inside `convert_char_fn::process_character()`:

```cpp
detail::character_flags_table_type flag =
code_point <= 0x00'FFFF ? d_flags[code_point] : 0; // ← hard-coded BMP cap
```

`d_flags` (`g_character_codepoint_flags`) and `d_case_table` (`g_character_cases_table`) are both 65,536-entry BMP-only arrays. Any `code_point > 0xFFFF` is hard-assigned `flag = 0`, so `IS_UPPER` / `IS_LOWER` / `IS_SPECIAL` are all false and the character is copied verbatim. The special-case mapping table (`d_special_case_mapping`) uses a perfect hash over 16-bit code points, so supplementary characters with multi-char mappings are equally unhandled.

All three tables need to be extended:

1. `g_character_codepoint_flags` — add `IS_UPPER` / `IS_LOWER` bits for supplementary cased code points.
2. `g_character_cases_table` — value type is `uint16_t` which cannot hold code points above U+FFFF; must be widened to `uint32_t` and supplementary entries added.
3. `g_special_case_mappings` / `get_special_case_hash_index` — the `codepoints_in` array in `char_cases.cu` is also `uint16_t`; both it and the prime search must be widened to `uint32_t` and the table regenerated.

Once the tables are extended the guard on line 142 of `case.cu` becomes an unconditional lookup (or the bound is raised to the full Unicode scalar range `<= 0x10FFFF`).

Affected functions: `to_upper`, `to_lower`, `swapcase`, and likely `capitalize` / `title` which share the same `convert_char_fn` path.

Affected Unicode scripts (non-exhaustive):

| Script | Range | Cased pairs |
|---|---|---|
| Deseret | U+10400–U+1044F | 40 |
| Osage | U+104B0–U+104D3 | 36 |
| Adlam | U+1E900–U+1E943 | 40 |
| Warang Citi | U+118A0–U+118DF | 64 |

**Environment overview**

- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
- Method of cuDF install: [conda, Docker, or from source]
- If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details.

**Additional context**

This is distinct from:
- **#21816** — fixed `utf8_to_codepoint` returning garbage for non-BMP input (parsing layer). This bug is one step later: the case-table lookup hard-discards non-BMP code points regardless of how correctly they were parsed.
- **#23005 / #23018** — Java JNI string encoding; a different layer entirely.

The `char_cases.cu` generator already exists and can be re-run after the tables are widened. The `codepoints_in` array in that file is the right starting point — it just needs supplementary entries added alongside the `uint16_t → uint32_t` widening.

Contributor guide

Open the contributing guide

Research direction

Run the Python reproduction first, then read cpp/src/strings/case.cu and the char_cases.cu generator, focusing on convert_char_fn::process_character and the named Unicode tables. Regenerate the tables with supplementary-plane entries and verify to_upper, to_lower, swapcase, capitalize, and title produce the expected case-folded results for non-BMP characters without regressing BMP behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.