internetarchive / internetarchive/openlibrary
Improve title normalization for non-English titles
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
On imports, title normalization strips trivial articles such as English "the " or "an "; but not French "le ", "la ", "les ", "un ", "une "; Spanish "el ", "la ", "los ", "una ", "uno "; Italian "il ", etc. This causes the creation of huge numbers of redundant work records for non-English works (even without considering translated popular works).
### Evidence / Screenshot (if possible)
### Relevant url?
https://github.com/internetarchive/openlibrary/blob/19f2062f519ff3189d36ce23d252f1f5a2d36c92/openlibrary/catalog/utils/__init__.py#L248
### Steps to Reproduce
1. Go to https://openlibrary.org/search?q=bestias+allende&mode=everything
2. See all the redundant works that were not needed
* Actual: found 11 works
* Expected: should find just one work
### Details
- **Logged in (Y/N)?**
- **Browser type/version?**
- **Operating system?**
- **Environment (prod/dev/local)?** prod
### Proposal & Constraints
Remove checks for hard-coded articles. Use instead a list for each major language, possibly based on those at:
https://en.wiktionary.org/wiki/the#Translations
https://en.wiktionary.org/wiki/an#Translations
### Related files
### Stakeholders
Contributor guide
Research direction
Start in openlibrary/catalog/utils/__init__.py around line 248, where title normalization handles articles, and review the linked Wiktionary translation lists. Extend normalization for the cited French, Spanish, Italian, and other major-language articles, then verify the bestias allende search no longer produces redundant works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100