internetarchive / internetarchive/openlibrary
Work dumps include authors without a key
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 126
Description
Works in the works dump have authors without a key, that are not present in the api response.
### Evidence / Screenshot (if possible)
Authors of OL24527121W in the work dump: `"authors": [{"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239668A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239669A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239670A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239671A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239672A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239673A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}]`
api response:
`{"title": "Medicina: Aspectos Epidemiol\u00f3gicos, Cl\u00ednicos e Estrat\u00e9gicos de Tratamento", "authors": [{"author": {"key": "/authors/OL9239668A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239669A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239670A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239671A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239672A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239673A"}, "type": {"key": "/type/author_role"}}], "key": "/works/OL24527121W", "type": {"key": "/type/work"}, "latest_revision": 1, "revision": 1, "created": {"type": "/type/datetime", "value": "2021-05-25T16:10:52.471943"}, "last_modified": {"type": "/type/datetime", "value": "2021-05-25T16:10:52.471943"}}`
### Relevant url?
[Work API](https://openlibrary.org/works/OL24527121W.json)
### Steps to Reproduce
1. Read thourgh the work dump
2. Find that work in the work dump
3. Observe that authors array
4. Compare to the api response
### Related files
[script/oldump.sh](https://github.com/internetarchive/openlibrary/blob/ceca4ccca599ba4d2660d687937cdd85fc8b9a08/scripts/oldump.sh)
[script/oldump.py](https://github.com/internetarchive/openlibrary/blob/ceca4ccca599ba4d2660d687937cdd85fc8b9a08/scripts/oldump.py)
https://github.com/internetarchive/openlibrary/blob/ceca4ccca599ba4d2660d687937cdd85fc8b9a08/openlibrary/data/dump.py#L328
### Stakeholders
@cdrini
Contributor guide
Research direction
Start with scripts/oldump.sh, scripts/oldump.py, and openlibrary/data/dump.py around line 328. Reproduce the discrepancy for work OL24527121W by comparing the work dump with https://openlibrary.org/works/OL24527121W.json. Done means the dump's authors entries contain the same author keys as the API response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100