googlefonts / googlefonts/fontc
Difference between python unicodedata and icu4x BidiClass queries for private use and unmapped codepoints
- Dominant language
- Rust
- Stars
- 193
- Forks
- 21
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 65
Description
To start on a positive note, I've learned some interesting things today!
Let's focus on two codepoints, `U+0378` and `U+0590`. The first is in the private use block, and the second is an unassigned codepoint in the hebrew block.
In python, `unicodedata.bidrectional(c)` returns `None` for both of these codepoints.
In rus/icu4x, `CodePointMapData::::new().get(c)` returns `BidiClass:LeftToRight` for the first codepoint, and `RightToLeft` for the second.
Digging into the unicode docs, icu4x is right here. Although neither of these codepoints is mapped in [`UnicodeData.txt`](https://www.unicode.org/Public/latest/ucd/UnicodeData.txt), there are explicit default values provided for these properties (in [`DerivedBidiClass.txt`](https://www.unicode.org/Public/17.0.0/ucd/extracted/DerivedBidiClass.txt) and the spec is clear that those values should be used when no specific value is provided.
Given this, my default stance would be to fix the python. In this case, though, at least in theory one of our sources of truth is the `unicodedata` module that is included with python, and I'm not sure if there is a feasible approach to fixing _that_.
It would be possible to handle this on the python side with a wrapper fn that checked if the returned value is `None` and then performed a check against the derived list, and maybe that would be the best approach?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the discrepancy with Python's unicodedata.bidirectional(c) and rus/icu4x CodePointMapData::::new().get(c) for U+0378 and U+0590. Read UnicodeData.txt and DerivedBidiClass.txt to confirm the default-property rules. Done means reaching a documented, tested decision about whether and where the differing behavior should be corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- internationalization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100