internetarchive / internetarchive/openlibrary

Imports via `load()` appear to accept only one language

Open
#9,490 6 comments 0 reactions 0 assignees View on GitHub
Lead: @mekarpeles Module: Import Needs: Breakdown Priority: 3 Python Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

Imports via `load()` that specify multiple languages seem to only accept one language. It is unclear if this only applies to the `/api/import/ia` endpoint.

It should be tested with other endpoints, such as `/api/import`. See the [Developer's Guide to Data Importing](https://github.com/internetarchive/openlibrary/wiki/Developer's-Guide-to-Data-Importing) for the many fun ways to import via the various endpoints with various languages and tools.

#### Evidence / Screenshot
Consider the IA item [2e013a64-935f-4be4-9c51-3eac22929627](https://archive.org/details/2e013a64-935f-4be4-9c51-3eac22929627), which has [two languages](https://archive.org/metadata/2e013a64-935f-4be4-9c51-3eac22929627/metadata/language).

Importing it, with #9480 applied (and #9480 **MUST** be applied or already in `master`), works:
```
❯ curl -X POST "http://localhost:8080/api/import/ia" \
-b ~/cookies.txt \
-d "identifier=2e013a64-935f-4be4-9c51-3eac22929627&require_marc=false"
{"authors": [{"key": "/authors/OL12A", "name": "Ingrid Robeyns", "status": "created"}], "success": true, "edition": {"k
ey": "/books/OL22M", "status": "created"}, "work": {"key": "/works/OL8W", "status": "created"}}
```

However, once imported, it only has `English` as a language: `Spanish` has disappeared into the ether.

### Reproducing the bug

1. Try to import something via `/api/import/ia` that has multiple languages, such as `ocaid` `2e013a64-935f-4be4-9c51-3eac22929627`.

* Expected behavior: Both `English` and `Spanish` show up in the edition as languages.
* Actual behavior: Only `English` shows up.

### Context

- Browser (Chrome, Safari, Firefox, etc):
- OS (Windows, Mac, etc):
- Logged in (Y/N): Y
- Environment (prod, dev, local): prod

### Notes from this Issue's Lead

#### Proposal & constraints

#### Related files

This will take more research, but by the time the `rec` gets to `build_query` in `openlibrary/catalog/add_book/load_book.py`, the only language code is `ENG`: https://github.com/internetarchive/openlibrary/blob/c546c6ab62c6435796109ca6a8169a673b842c75/openlibrary/catalog/add_book/load_book.py#L288-L324

You'll have to trust me, but this is from a `print(f"{rec = }", flush=True)` called just after the docstring in `build_query`:
```python
rec = {'title': 'Tener Demasiado', 'authors': [{'name': 'Ingrid Robeyns'}], 'publish_date': '2024-02-19', 'description': "'Tener demasiado' es el primer volumen académico dedicado al limitarismo: la idea de que el uso de los recursos económicos o de los ecosistemas
no sobrepasen ciertos límites. \n\nSe trata de un concepto profundamente arraigado en el pensamiento económico y político, por lo que es posible encontrar premisas similares en pensadores como Platón, Aquino o Spinoza. No obstante, 'Tener demasiado' es el primer ejemplar en el campo
de la filosofía política contemporánea en el que el limitarismo se explora en profundidad y con detalle.\n\nAsimismo, este estudio reúne por primera vez los mejores escritos de los principales teóricos del limitarismo, lo que le convierte en una contribución esencial al campo de la f
ilosofía política, en general, y de las teorías sobre la justicia distributiva, en particular. Incluye tanto artículos seminales ya publicados como nuevos capítulos y se presenta como lectura indispensable para académicos y estudiantes de teoría política y filosofía, así como para to
dos aquellos interesados en cuestiones relacionadas con la justicia distributiva.", 'isbn_13': ['9781805110804', '9781805110811', '9781805110828', '9781805110866', '9781805110835'], 'languages': ['ENG'], 'subjects': ['HP', 'HPCF', 'HPS', 'KCA', 'RNA', 'PHI000000', 'PHI019000', 'PHI03
4000', 'POL023000', 'POL044000', 'KCP', 'QD', 'QDTS', 'RND', 'Economics, Politics and Sociology', 'Other languages', 'Philosophy', 'Generaciones futuras', 'Justicia distributiva', 'Justicia intergeneracional', 'Limitarismo', 'Limitarismo ecológico', 'Limitarismo económico', 'Recursos
materiales'], 'oclc': ['1422929642'], 'number_of_pages': 458, 'publishers': ['Open Book Publishers'], 'ocaid': '2e013a64-935f-4be4-9c51-3eac22929627', 'source_records': ['ia:2e013a64-935f-4be4-9c51-3eac22929627'], 'subtitle': 'Ensayos Filosóficos sobre el Limitarismo'}
```

#### Stakeholders


#### Instructions for Contributors

- Please note this **MUST** have #9488 applied if has not already been merged to `master`. Without it, the import via `/api/import/ia` will fail.
- Please [run these commands](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://github.com/internetarchive/openlibrary/wiki/Git-Cheat-Sheet#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start in openlibrary/catalog/add_book/load_book.py, especially build_query around the point where the issue reports that rec['languages'] contains only ENG. Reproduce the /api/import/ia case with OCAID 2e013a64-935f-4be4-9c51-3eac22929627 after applying #9488, then compare the other import endpoints described in the Developer's Guide. Done means imports preserve both English and Spanish languages.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.