internetarchive / internetarchive/openlibrary
Imports via `load()` appear to accept only one language
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Problem
Imports via `load()` that specify multiple languages seem to only accept one language. It is unclear if this only applies to the `/api/import/ia` endpoint.
It should be tested with other endpoints, such as `/api/import`. See the [Developer's Guide to Data Importing](https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing) for the many fun ways to import via the various endpoints with various languages and tools.
#### Evidence / Screenshot
Consider the IA item [2e013a64-935f-4be4-9c51-3eac22929627](https://archive.org/details/2e013a64-935f-4be4-9c51-3eac22929627), which has [two languages](https://archive.org/metadata/2e013a64-935f-4be4-9c51-3eac22929627/metadata/language).
Importing it, with #9480 applied (and #9480 **MUST** be applied or already in `master`), works:
```
❯ curl -X POST "http://localhost:8080/api/import/ia" \
-b ~/cookies.txt \
-d "identifier=2e013a64-935f-4be4-9c51-3eac22929627&require_marc=false"
{"authors": [{"key": "/authors/OL12A", "name": "Ingrid Robeyns", "status": "created"}], "success": true, "edition": {"k
ey": "/books/OL22M", "status": "created"}, "work": {"key": "/works/OL8W", "status": "created"}}
```
However, once imported, it only has `English` as a language: `Spanish` has disappeared into the ether.
### Reproducing the bug
1. Try to import something via `/api/import/ia` that has multiple languages, such as `ocaid` `2e013a64-935f-4be4-9c51-3eac22929627`.
* Expected behavior: Both `English` and `Spanish` show up in the edition as languages.
* Actual behavior: Only `English` shows up.
### Context
- Browser (Chrome, Safari, Firefox, etc):
- OS (Windows, Mac, etc):
- Logged in (Y/N): Y
- Environment (prod, dev, local): prod
### Notes from this Issue's Lead
#### Proposal & constraints
#### Related files
This will take more research, but by the time the `rec` gets to `build_query` in `openlibrary/catalog/add_book/load_book.py`, the only language code is `ENG`: https://github.com/internetarchive/openlibrary/blob/c546c6ab62c6435796109ca6a8169a673b842c75/openlibrary/catalog/add_book/load_book.py#L288-L324
You'll have to trust me, but this is from a `print(f"{rec = }", flush=True)` called just after the docstring in `build_query`:
```python
rec = {'title': 'Tener Demasiado', 'authors': [{'name': 'Ingrid Robeyns'}], 'publish_date': '2024-02-19', 'description': "'Tener demasiado' es el primer volumen académico dedicado al limitarismo: la idea de que el uso de los recursos económicos o de los ecosistemas
no sobrepasen ciertos límites. \n\nSe trata de un concepto profundamente arraigado en el pensamiento económico y político, por lo que es posible encontrar premisas similares en pensadores como Platón, Aquino o Spinoza. No obstante, 'Tener demasiado' es el primer ejemplar en el campo
de la filosofía política contemporánea en el que el limitarismo se explora en profundidad y con detalle.\n\nAsimismo, este estudio reúne por primera vez los mejores escritos de los principales teóricos del limitarismo, lo que le convierte en una contribución esencial al campo de la f
ilosofía política, en general, y de las teorías sobre la justicia distributiva, en particular. Incluye tanto artículos seminales ya publicados como nuevos capítulos y se presenta como lectura indispensable para académicos y estudiantes de teoría política y filosofía, así como para to
dos aquellos interesados en cuestiones relacionadas con la justicia distributiva.", 'isbn_13': ['9781805110804', '9781805110811', '9781805110828', '9781805110866', '9781805110835'], 'languages': ['ENG'], 'subjects': ['HP', 'HPCF', 'HPS', 'KCA', 'RNA', 'PHI000000', 'PHI019000', 'PHI03
4000', 'POL023000', 'POL044000', 'KCP', 'QD', 'QDTS', 'RND', 'Economics, Politics and Sociology', 'Other languages', 'Philosophy', 'Generaciones futuras', 'Justicia distributiva', 'Justicia intergeneracional', 'Limitarismo', 'Limitarismo ecológico', 'Limitarismo económico', 'Recursos
materiales'], 'oclc': ['1422929642'], 'number_of_pages': 458, 'publishers': ['Open Book Publishers'], 'ocaid': '2e013a64-935f-4be4-9c51-3eac22929627', 'source_records': ['ia:2e013a64-935f-4be4-9c51-3eac22929627'], 'subtitle': 'Ensayos Filosóficos sobre el Limitarismo'}
```
#### Stakeholders
#### Instructions for Contributors
- Please note this **MUST** have #9488 applied if has not already been merged to `master`. Without it, the import via `/api/import/ia` will fail.
- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start in openlibrary/catalog/add_book/load_book.py, especially build_query around the point where the issue reports that rec['languages'] contains only ENG. Reproduce the /api/import/ia case with OCAID 2e013a64-935f-4be4-9c51-3eac22929627 after applying #9488, then compare the other import endpoints described in the Developer's Guide. Done means imports preserve both English and Spanish languages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100