opensanctions / opensanctions/poliloom

Import should promote matching extracted properties instead of creating duplicates

Open
#137 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

loom
Dominant language
Python
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

After the PropertyReference refactoring, dedup between extracted and imported properties only works in one direction.

Extraction → existing property (works): When enrichment.py extracts a property, it calls Property.find_matching() to check if a matching property already exists (whether from Wikidata or a previous extraction). If found, it calls add_reference() to attach the new source to the existing property instead of creating a duplicate.

Import → existing extracted property (broken): When a Wikidata re-import brings in a statement that matches an existing extracted property, the importer creates a new Property row. This happens because the batch upsert conflicts on statement_id, and extracted properties have statement_id = NULL — so there's no conflict, and a new row is inserted. Previously soft_delete_matching_extracted() handled this, but it was removed in the PropertyReference refactoring.

The result: if a politician has an extracted property (e.g. birth date = 1960, no statement_id) and Wikidata later adds a statement for that same fact, you end up with two Property rows for the same fact.

Proposed fix

Use find_matching() during import as well. In _insert_politicians_batch, before batch-upserting each property, check for a matching extracted property. If one exists, promote it: set statement_id and references_json on the existing row. This preserves its PropertyReferences (the extraction sources) while marking it as Wikidata-backed. Promoted properties are then excluded from the batch_upsert.

This makes find_matching() the single generic dedup mechanism used in both directions:

  • Extraction finds existing Wikidata property → add_reference()
  • Import finds existing extracted property → promote it (set statement_id, references_json)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in enrichment.py and trace the import path through _insert_politicians_batch, then inspect Property.find_matching() and the batch upsert behavior. Done means a matching extracted property is promoted without creating a duplicate, its existing PropertyReferences remain intact, and promoted properties are excluded from the batch upsert.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.