internetarchive / internetarchive/openlibrary

Improve title normalization for non-English titles

Open
#2,919 1 comment 0 reactions 0 assignees View on GitHub
Lead: @cdrini Module: Import Priority: 3 Theme: Internationalization (i18n) Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

On imports, title normalization strips trivial articles such as English "the " or "an "; but not French "le ", "la ", "les ", "un ", "une "; Spanish "el ", "la ", "los ", "una ", "uno "; Italian "il ", etc. This causes the creation of huge numbers of redundant work records for non-English works (even without considering translated popular works).

### Evidence / Screenshot (if possible)

### Relevant url?

https://github.com/internetarchive/openlibrary/blob/19f2062f519ff3189d36ce23d252f1f5a2d36c92/openlibrary/catalog/utils/__init__.py#L248

### Steps to Reproduce

1. Go to https://openlibrary.org/search?q=bestias+allende&mode=everything
2. See all the redundant works that were not needed

* Actual: found 11 works
* Expected: should find just one work

### Details

- **Logged in (Y/N)?**
- **Browser type/version?**
- **Operating system?**
- **Environment (prod/dev/local)?** prod

### Proposal & Constraints

Remove checks for hard-coded articles. Use instead a list for each major language, possibly based on those at:
https://en.wiktionary.org/wiki/the#Translations
https://en.wiktionary.org/wiki/an#Translations

### Related files

### Stakeholders

Contributor guide

Open the contributing guide

Research direction

Start in openlibrary/catalog/utils/__init__.py around line 248, where title normalization handles articles, and review the linked Wiktionary translation lists. Extend normalization for the cited French, Spanish, Italian, and other major-language articles, then verify the bestias allende search no longer produces redundant works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.