internetarchive / internetarchive/openlibrary

Convert romanized cyrllic text to cyrllic alphabet

Open
#10,823 10 comments 0 reactions 0 assignees View on GitHub
Affects: Data Lead: @cdrini Priority: 2 State: Blocked Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

There are many records on Open Library where Cyrllic titles have been saved into the catalogue with their romanized equivalents, eg "Nafta Skhidnoï Halychyny" instead of "Нафта Східноï Галичини" (https://openlibrary.org/works/OL15708029W/Nafta_Skhidno%C3%AF_Halychyny?edition=key%3A/books/OL24631439M )

This colab (from like 2023 :P) contains a solution to fix them and save them back to Open Library: https://colab.research.google.com/drive/1baJbipaVKarkEAkdcjcnxTZgdGprPJMm

The last run yielded this spreadsheet, which was evaluated by folks with better knowledge of Cyrillic scripts than myself! And yielded these comments: https://docs.google.com/spreadsheets/d/17NUtyK9oo0BpMXrrDkmQS_hv7ezmZzbOdjm8B3Az_Rg/edit#gid=1778845672

A few outstanding issues:

- [x] Certain characters not being converted correctly (`ĭ`, `ï`)
- [ ] English/latin text is sometimes being converted; can we detect somehow? Or is it happening infrequently enough that we deal with the error?

Next step: Run the colab once again since the above is fixed, and put the resulting preview of the edits in the spreadsheet so that we can human verify again, and determine the prevalence of the English/Latin text issue, or possibly find a fix.

### Reproducing the bug

NA

### Context

NA

### Breakdown

#### Requirements Checklist
* [ ]

#### Related files

*

#### Stakeholders

@bicolino34

#### Instructions for Contributors

- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Colab and the existing spreadsheet of reviewed Cyrillic conversions. Run the notebook again with the character fixes, inspect cases where English or Latin text is converted, and add the resulting edit preview to the spreadsheet. Done means the new results are available for human verification and the prevalence or handling of false conversions is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
content, data, localization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.