opensanctions / opensanctions/poliloom
Import should promote matching extracted properties instead of creating duplicates
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Problem
After the PropertyReference refactoring, dedup between extracted and imported properties only works in one direction.
Extraction → existing property (works): When enrichment.py extracts a property, it calls Property.find_matching() to check if a matching property already exists (whether from Wikidata or a previous extraction). If found, it calls add_reference() to attach the new source to the existing property instead of creating a duplicate.
Import → existing extracted property (broken): When a Wikidata re-import brings in a statement that matches an existing extracted property, the importer creates a new Property row. This happens because the batch upsert conflicts on statement_id, and extracted properties have statement_id = NULL — so there's no conflict, and a new row is inserted. Previously soft_delete_matching_extracted() handled this, but it was removed in the PropertyReference refactoring.
The result: if a politician has an extracted property (e.g. birth date = 1960, no statement_id) and Wikidata later adds a statement for that same fact, you end up with two Property rows for the same fact.
Proposed fix
Use find_matching() during import as well. In _insert_politicians_batch, before batch-upserting each property, check for a matching extracted property. If one exists, promote it: set statement_id and references_json on the existing row. This preserves its PropertyReferences (the extraction sources) while marking it as Wikidata-backed. Promoted properties are then excluded from the batch_upsert.
This makes find_matching() the single generic dedup mechanism used in both directions:
- Extraction finds existing Wikidata property →
add_reference() - Import finds existing extracted property → promote it (set
statement_id,references_json)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in enrichment.py and trace the import path through _insert_politicians_batch, then inspect Property.find_matching() and the batch upsert behavior. Done means a matching extracted property is promoted without creating a duplicate, its existing PropertyReferences remain intact, and promoted properties are excluded from the batch upsert.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 64/100