internetarchive / internetarchive/openlibrary
Convert romanized cyrllic text to cyrllic alphabet
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Problem
There are many records on Open Library where Cyrllic titles have been saved into the catalogue with their romanized equivalents, eg "Nafta Skhidnoï Halychyny" instead of "Нафта Східноï Галичини" (https://openlibrary.org/works/OL15708029W/Nafta_Skhidno%C3%AF_Halychyny?edition=key%3A/books/OL24631439M )
This colab (from like 2023 :P) contains a solution to fix them and save them back to Open Library: https://colab.research.google.com/drive/1baJbipaVKarkEAkdcjcnxTZgdGprPJMm
The last run yielded this spreadsheet, which was evaluated by folks with better knowledge of Cyrillic scripts than myself! And yielded these comments: https://docs.google.com/spreadsheets/d/17NUtyK9oo0BpMXrrDkmQS_hv7ezmZzbOdjm8B3Az_Rg/edit#gid=1778845672
A few outstanding issues:
- [x] Certain characters not being converted correctly (`ĭ`, `ï`)
- [ ] English/latin text is sometimes being converted; can we detect somehow? Or is it happening infrequently enough that we deal with the error?
Next step: Run the colab once again since the above is fixed, and put the resulting preview of the edits in the spreadsheet so that we can human verify again, and determine the prevalence of the English/Latin text issue, or possibly find a fix.
### Reproducing the bug
NA
### Context
NA
### Breakdown
#### Requirements Checklist
* [ ]
#### Related files
*
#### Stakeholders
@bicolino34
#### Instructions for Contributors
- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start with the linked Colab and the existing spreadsheet of reviewed Cyrillic conversions. Run the notebook again with the character fixes, inspect cases where English or Latin text is converted, and add the resulting edit preview to the spreadsheet. Done means the new results are available for human verification and the prevalence or handling of false conversions is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- content, data, localization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100