internetarchive / internetarchive/openlibrary
Unicode: transcoding errors for \u0361
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
reported on Launchpad https://bugs.launchpad.net/openlibrary/+bug/1020006
There are a lot of transcoding errors from imported external data regarding \u0361 LIGATURE TIE/COMBINING DOUBLE INVERTED BREVE. Ligature tie is used in romanization of Cyrillic.
External data is often using encodings where U+0361 is split into U+FE20 and U+FE21:
- i︠a︡ [i + U+FE20 + a + U+FE21] should be converted to i͡a [i + U+0361 + a].
Furthermore if there is a dot [U+307] above the ligature tie [U+0361], there should be a CGJ U+034F to prevent Unicode canonical re-ordering:
- t︠̇s︡ [t + U+FE20 + U+0307 + s + U+FE21] should be converted to t͡͏̇s [t + U+0361 + U+034F + U+0307 + s].
See the examples in section 7.9 of http://www.unicode.org/versions/Unicode6.1.0/ch07.pdf
But the problem is external data often contains errors instead.
1. U+FE20 has apparently been converted to U+0361 but U+FE21 is still there.
See this fix for example
http://openlibrary.org/books/OL727005M/Zvukovo%C4%AD_sostav_chechenskogo_literaturnogo_i%CD%A1azyka_i_voprosy_usovershenstvovanii%CD%A1a_alfavita_i_orfogr?m=diff&b=4
1. U+FE20 and U+FE21 have not been converted to U+0361.
2. U+FE20, U+FE21 and other diacritic have not been converted to U+0361 + CGJ + other diacritic
3. U+0360 is used instead of U+0361
See http://openlibrary.org/works/OL1070139W which seems to have a bit of everything.
I think most of the errors can be fixed they seem to follow these patterns and they should match a specific set of transliteration digraphs.
Contributor guide
Assessment
This issue has not been assessed yet.