opensanctions / opensanctions/poliloom

Importer creates duplicate citizenship properties for same country with different time periods

Open
#113 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

loom
Dominant language
Python
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

The importer creates multiple citizenship properties for the same politician-country combination when Wikidata has multiple statements with different start/end time qualifiers. This results in 30 duplicate entries in our database.

Examples

San Marino (Q238) is particularly affected with 7 politicians having duplicate citizenships.

Expected Behavior

We should only store one citizenship property per politician-country combination. The time period qualifiers (start/end dates) are not relevant for our use case.

Suggested Fix

During import, deduplicate citizenship statements by (politician_id, entity_id) - only import the first one encountered or the one without time qualifiers.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the importer code that processes Wikidata citizenship statements and reproduce the duplicate cases listed in the issue. Ensure the import keeps one citizenship per politician-country pair regardless of time qualifiers, then verify that the affected examples no longer create duplicate database entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.