internetarchive / internetarchive/openlibrary

DNB duplicated edition import

Open
#13,656 2 comments 0 reactions 0 assignees View on GitHub
Affects: Data Module: Import Needs: Breakdown Needs: Investigation Needs: Lead Needs: Response Needs: Triage Theme: MARC records Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Problem

Re-importing this one MARC record matches the Author and Work (as expected), but always seems to create a new edition (when multiple seemingly identical editions exist). Why? It seems repeatable.

**Most recent edition import that created _another_ edition record:** https://openlibrary.org/books/OL62559907M

**Matched work:** https://openlibrary.org/works/OL28180241W

**Source MARC record:** https://openlibrary.org/show-records/marc_dnb_202006/dnb_all_dnbmarc_20200615-3.mrc:17091:1027

Existing editions, all from apparently the same source with same DNB identifier:
* https://openlibrary.org/books/OL38578338M.yml
* https://openlibrary.org/books/OL49838858M.yml

### Preview test import:
```bash
../bulk-import.py -p -n1 -o17091 -f dnb_all_dnbmarc_20200615-3.mrc marc_dnb_202006
```

```
marc_dnb_202006/dnb_all_dnbmarc_20200615-3.mrc:17091:5: 200 --
{'preview': True,
'edits': [{'type': {'key': '/type/edition'}, 'publish_date': '2010', 'publish_country': 'gw', 'languages': [{'key': '/languages/ger'}], 'title': 'Begleitfehlbildungen bei kongenitaler Zwerchfellhernie', 'by_statement': 'vorgelegt von Katharina Link', 'identifiers': {'dnb': ['1013074777']}, 'authors': [{'key': '/authors/OL10499570A'}], 'dewey_decimal_class': ['610'], 'notes': {'type': '/type/text', 'value': 'Heidelberg, Univ., Diss., 2011'}, 'pagination': 'VII, 91 Bl.', 'number_of_pages': {'type': '/type/int', 'value': 91}, 'source_records': ['marc:marc_dnb_202006/dnb_all_dnbmarc_20200615-3.mrc:17091:1027'], 'full_title': 'Begleitfehlbildungen bei kongenitaler Zwerchfellhernie', 'works': [{'key': '/works/OL28180241W'}], 'key': '/books/__new__1acd28f1-1f46-48ed-b890-192a5cf3e141'}],
authors': [{'key': '/authors/OL10499570A', 'name': 'Katharina Link', 'status': 'matched'}], 'success': True,
'edition': {'key': '/books/__new__1acd28f1-1f46-48ed-b890-192a5cf3e141', 'status': 'created'},
'work': {'key': '/works/OL28180241W', 'status': 'matched'},
'next_record_offset': 18118, 'next_record_length': 8
```

The expectation is that given editions were already created, the json for the edition status will be something like
```json
'edition': {'key': '/books/__new__f9d85cb4-c159-477e-b6b4-533dc358d5b5', 'status': 'matched'}
```
(rather than `status: created`)

Most basic hypothesis: Search index had somehow _missed_ this in the past. Presumably if the above import command succeeds after the time it takes the new OL62559907M to be indexed, that was the problem. If the command continues to create duplicates, there is a more systematic problem. Either with the matching code, or possibly with the indexing code? (maybe there is something about this record that prevents it from being indexed correctly?)

### Context

Testing the CLI command requires an authorized bot account.

### Breakdown

Implementation Details (for maintainers)

#### Requirements Checklist
* [ ]

#### Related files

*

#### Stakeholders

*


#### Instructions for Contributors

- Please [run these commands](https://docs.openlibrary.org/developers/tools/git.html#working-on-your-branch) to ensure your repository is up to date **before** [creating a new branch](https://docs.openlibrary.org/developers/tools/git.html#making-changes-and-creating-a-pull-request) to work on this issue and **each time after** pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

Contributor guide

Open the contributing guide

Research direction

Start with bulk-import.py and reproduce the preview command against the cited DNB MARC record. Inspect how edition matching and indexing handle the existing DNB identifier, then repeat the import after OL62559907M has had time to index. Done means the preview reports the edition as matched rather than created, and the cause of any continued duplication is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.