internetarchive / internetarchive/openlibrary
MARC imports with incorrect character encoding interpretation
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Problem
This search: https://openlibrary.org/search/authors?q=Odo+&sort=name
shows a bunch of recent MARC imports with corrupt Chinese language strings.
As an example https://openlibrary.org/books/OL60300732M was imported as
̆ ̈ ̄ ̄(1999-2010)
when the MARC record lists the title as 澄迈县志(1999-2010)
The imports seem to span about the last year and a half or so, but not sure of the exact date boundaries
### Reproducing the bug
* Expected behavior: Titles and names of authors are imported correctly
* Actual behavior: Titles and names are imported as mojibake
### Context
- Browser (Chrome, Safari, Firefox, etc): Chrome
- OS (Windows, Mac, etc): Mac
- Logged in (Y/N): Y
- Environment (prod, dev, local): prod
### Breakdown
Implementation Details (for maintainers)
#### Requirements Checklist
* [ ]
#### Related files
*
#### Stakeholders
* @hornc
#### Instructions for Contributors
- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start by tracing the MARC import path and reproducing the example for OL60300732M, comparing the imported title with the MARC record's 澄迈县志(1999-2010). Identify where character encoding is interpreted during import; done means titles and author names, including Chinese strings, are imported correctly without mojibake.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100