googlefonts / googlefonts/fontc

Difference between python unicodedata and icu4x BidiClass queries for private use and unmapped codepoints

Open
#1,782 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
193
Forks
21
Avg merge
1d 17h
Merged PRs (30d)
65

Description

To start on a positive note, I've learned some interesting things today!

Let's focus on two codepoints, `U+0378` and `U+0590`. The first is in the private use block, and the second is an unassigned codepoint in the hebrew block.

In python, `unicodedata.bidrectional(c)` returns `None` for both of these codepoints.

In rus/icu4x, `CodePointMapData::::new().get(c)` returns `BidiClass:LeftToRight` for the first codepoint, and `RightToLeft` for the second.

Digging into the unicode docs, icu4x is right here. Although neither of these codepoints is mapped in [`UnicodeData.txt`](https://www.unicode.org/Public/latest/ucd/UnicodeData.txt), there are explicit default values provided for these properties (in [`DerivedBidiClass.txt`](https://www.unicode.org/Public/17.0.0/ucd/extracted/DerivedBidiClass.txt) and the spec is clear that those values should be used when no specific value is provided.

Given this, my default stance would be to fix the python. In this case, though, at least in theory one of our sources of truth is the `unicodedata` module that is included with python, and I'm not sure if there is a feasible approach to fixing _that_.

It would be possible to handle this on the python side with a wrapper fn that checked if the returned value is `None` and then performed a check against the derived list, and maybe that would be the best approach?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the discrepancy with Python's unicodedata.bidirectional(c) and rus/icu4x CodePointMapData::::new().get(c) for U+0378 and U+0590. Read UnicodeData.txt and DerivedBidiClass.txt to confirm the default-property rules. Done means reaching a documented, tested decision about whether and where the differing behavior should be corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
internationalization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.