internetarchive / internetarchive/openlibrary

MARC imports with incorrect character encoding interpretation

Open
#13,617 4 comments 0 reactions 0 assignees View on GitHub
Affects: Data Lead: @hornc Module: Import Needs: Breakdown Needs: Investigation Needs: Response Needs: Staff Decision Needs: Triage Theme: MARC records Theme: Unicode Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

This search: https://openlibrary.org/search/authors?q=Odo+&sort=name
shows a bunch of recent MARC imports with corrupt Chinese language strings.

As an example https://openlibrary.org/books/OL60300732M was imported as
̆ ̈ ̄ ̄(1999-2010)
when the MARC record lists the title as 澄迈县志(1999-2010)

The imports seem to span about the last year and a half or so, but not sure of the exact date boundaries

### Reproducing the bug

* Expected behavior: Titles and names of authors are imported correctly
* Actual behavior: Titles and names are imported as mojibake

### Context

- Browser (Chrome, Safari, Firefox, etc): Chrome
- OS (Windows, Mac, etc): Mac
- Logged in (Y/N): Y
- Environment (prod, dev, local): prod

### Breakdown

Implementation Details (for maintainers)

#### Requirements Checklist
* [ ]

#### Related files

*

#### Stakeholders

* @hornc


#### Instructions for Contributors

- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the MARC import path and reproducing the example for OL60300732M, comparing the imported title with the MARC record's 澄迈县志(1999-2010). Identify where character encoding is interpreted during import; done means titles and author names, including Chinese strings, are imported correctly without mojibake.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.