internetarchive / internetarchive/openlibrary

Unicode: transcoding errors for \u0361

Open
#150 16 comments 0 reactions 0 assignees View on GitHub
Affects: Data Lead: @mekarpeles Module: Import Priority: 3 Theme: Unicode Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

reported on Launchpad https://bugs.launchpad.net/openlibrary/+bug/1020006

There are a lot of transcoding errors from imported external data regarding \u0361 LIGATURE TIE/COMBINING DOUBLE INVERTED BREVE. Ligature tie is used in romanization of Cyrillic.

External data is often using encodings where U+0361 is split into U+FE20 and U+FE21:
- i︠a︡ [i + U+FE20 + a + U+FE21] should be converted to i͡a [i + U+0361 + a].

Furthermore if there is a dot [U+307] above the ligature tie [U+0361], there should be a CGJ U+034F to prevent Unicode canonical re-ordering:
- t︠̇s︡ [t + U+FE20 + U+0307 + s + U+FE21] should be converted to t͡͏̇s [t + U+0361 + U+034F + U+0307 + s].
See the examples in section 7.9 of http://www.unicode.org/versions/Unicode6.1.0/ch07.pdf

But the problem is external data often contains errors instead.
1. U+FE20 has apparently been converted to U+0361 but U+FE21 is still there.

See this fix for example
http://openlibrary.org/books/OL727005M/Zvukovo%C4%AD_sostav_chechenskogo_literaturnogo_i%CD%A1azyka_i_voprosy_usovershenstvovanii%CD%A1a_alfavita_i_orfogr?m=diff&b=4
1. U+FE20 and U+FE21 have not been converted to U+0361.
2. U+FE20, U+FE21 and other diacritic have not been converted to U+0361 + CGJ + other diacritic
3. U+0360 is used instead of U+0361

See http://openlibrary.org/works/OL1070139W which seems to have a bit of everything.

I think most of the errors can be fixed they seem to follow these patterns and they should match a specific set of transliteration digraphs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.