internetarchive / internetarchive/openlibrary
DNB duplicated edition import
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
### Problem
Re-importing this one MARC record matches the Author and Work (as expected), but always seems to create a new edition (when multiple seemingly identical editions exist). Why? It seems repeatable.
**Most recent edition import that created _another_ edition record:** https://openlibrary.org/books/OL62559907M
**Matched work:** https://openlibrary.org/works/OL28180241W
**Source MARC record:** https://openlibrary.org/show-records/marc_dnb_202006/dnb_all_dnbmarc_20200615-3.mrc:17091:1027
Existing editions, all from apparently the same source with same DNB identifier:
* https://openlibrary.org/books/OL38578338M.yml
* https://openlibrary.org/books/OL49838858M.yml
### Preview test import:
```bash
../bulk-import.py -p -n1 -o17091 -f dnb_all_dnbmarc_20200615-3.mrc marc_dnb_202006
```
```
marc_dnb_202006/dnb_all_dnbmarc_20200615-3.mrc:17091:5: 200 --
{'preview': True,
'edits': [{'type': {'key': '/type/edition'}, 'publish_date': '2010', 'publish_country': 'gw', 'languages': [{'key': '/languages/ger'}], 'title': 'Begleitfehlbildungen bei kongenitaler Zwerchfellhernie', 'by_statement': 'vorgelegt von Katharina Link', 'identifiers': {'dnb': ['1013074777']}, 'authors': [{'key': '/authors/OL10499570A'}], 'dewey_decimal_class': ['610'], 'notes': {'type': '/type/text', 'value': 'Heidelberg, Univ., Diss., 2011'}, 'pagination': 'VII, 91 Bl.', 'number_of_pages': {'type': '/type/int', 'value': 91}, 'source_records': ['marc:marc_dnb_202006/dnb_all_dnbmarc_20200615-3.mrc:17091:1027'], 'full_title': 'Begleitfehlbildungen bei kongenitaler Zwerchfellhernie', 'works': [{'key': '/works/OL28180241W'}], 'key': '/books/__new__1acd28f1-1f46-48ed-b890-192a5cf3e141'}],
authors': [{'key': '/authors/OL10499570A', 'name': 'Katharina Link', 'status': 'matched'}], 'success': True,
'edition': {'key': '/books/__new__1acd28f1-1f46-48ed-b890-192a5cf3e141', 'status': 'created'},
'work': {'key': '/works/OL28180241W', 'status': 'matched'},
'next_record_offset': 18118, 'next_record_length': 8
```
The expectation is that given editions were already created, the json for the edition status will be something like
```json
'edition': {'key': '/books/__new__f9d85cb4-c159-477e-b6b4-533dc358d5b5', 'status': 'matched'}
```
(rather than `status: created`)
Most basic hypothesis: Search index had somehow _missed_ this in the past. Presumably if the above import command succeeds after the time it takes the new OL62559907M to be indexed, that was the problem. If the command continues to create duplicates, there is a more systematic problem. Either with the matching code, or possibly with the indexing code? (maybe there is something about this record that prevents it from being indexed correctly?)
### Context
Testing the CLI command requires an authorized bot account.
### Breakdown
Implementation Details (for maintainers)
#### Requirements Checklist
* [ ]
#### Related files
*
#### Stakeholders
*
#### Instructions for Contributors
- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
Contributor guide
Research direction
Start with bulk-import.py and reproduce the preview command against the cited DNB MARC record. Inspect how edition matching and indexing handle the existing DNB identifier, then repeat the import after OL62559907M has had time to index. Done means the preview reports the edition as matched rather than created, and the cause of any continued duplication is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100