internetarchive / internetarchive/openlibrary

Work dumps include authors without a key

Open
#8,834 3 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Priority: 3 Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 14h
Merged PRs (30d)
126

Description

Works in the works dump have authors without a key, that are not present in the api response.
### Evidence / Screenshot (if possible)
Authors of OL24527121W in the work dump: `"authors": [{"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239668A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239669A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239670A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239671A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239672A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239673A"}, "type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}, {"type": {"key": "/type/author_role"}}]`
api response:
`{"title": "Medicina: Aspectos Epidemiol\u00f3gicos, Cl\u00ednicos e Estrat\u00e9gicos de Tratamento", "authors": [{"author": {"key": "/authors/OL9239668A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239669A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239670A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239671A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239672A"}, "type": {"key": "/type/author_role"}}, {"author": {"key": "/authors/OL9239673A"}, "type": {"key": "/type/author_role"}}], "key": "/works/OL24527121W", "type": {"key": "/type/work"}, "latest_revision": 1, "revision": 1, "created": {"type": "/type/datetime", "value": "2021-05-25T16:10:52.471943"}, "last_modified": {"type": "/type/datetime", "value": "2021-05-25T16:10:52.471943"}}`
### Relevant url?

[Work API](https://openlibrary.org/works/OL24527121W.json)
### Steps to Reproduce

1. Read thourgh the work dump
2. Find that work in the work dump
3. Observe that authors array
4. Compare to the api response

### Related files

[script/oldump.sh](https://github.com/internetarchive/openlibrary/blob/ceca4ccca599ba4d2660d687937cdd85fc8b9a08/scripts/oldump.sh)
[script/oldump.py](https://github.com/internetarchive/openlibrary/blob/ceca4ccca599ba4d2660d687937cdd85fc8b9a08/scripts/oldump.py)
https://github.com/internetarchive/openlibrary/blob/ceca4ccca599ba4d2660d687937cdd85fc8b9a08/openlibrary/data/dump.py#L328
### Stakeholders

@cdrini

Contributor guide

Open the contributing guide

Research direction

Start with scripts/oldump.sh, scripts/oldump.py, and openlibrary/data/dump.py around line 328. Reproduce the discrepancy for work OL24527121W by comparing the work dump with https://openlibrary.org/works/OL24527121W.json. Done means the dump's authors entries contain the same author keys as the API response.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.