opensanctions / opensanctions/poliloom

API uses naive [0] index for multiple P580/P582 qualifiers in timeframe comparison

Open
#102 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

loom
Dominant language
Python
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

When comparing position timeframes during import/enrichment, the API extracts only the first qualifier value ([0]) for P580 (start date) and P582 (end date). This is problematic because ~389 positions have multiple P580 values and similar numbers have multiple P582 values.

Affected Code

poliloom/models/politician.py:934-960:

@staticmethod
def _extract_timeframe_from_qualifiers(qualifiers_json: dict | None):
    if "P580" in qualifiers_json:
        start_data = qualifiers_json["P580"][0]["datavalue"]["value"]  # <-- [0] only
        start_date = WikidataDate.from_wikidata_time(...)
    if "P582" in qualifiers_json:
        end_data = qualifiers_json["P582"][0]["datavalue"]["value"]  # <-- [0] only

This method is used by:

  • Property._compare_to() - comparing properties during import
  • Property.should_store() - deciding whether to store new properties
  • Property.soft_delete_matching_extracted() - deduplication during import

Additionally, Property.format_timeframe() (lines 893-931) also uses [0] for display purposes.

Impact

Incorrect timeframe extraction can cause:

  1. False matches - positions incorrectly identified as duplicates
  2. Missed matches - overlapping positions not detected
  3. Inconsistent display - arbitrary date shown instead of most representative

GUI Solution (for reference)

The GUI solved this in poliloom-gui/src/lib/wikidata/qualifierParser.ts with selectBestDate():

  1. Parse all date values from the qualifier array
  2. Sort by precision (most precise first: day > month > year)
  3. For equal precision: earliest for P580, latest for P582
  4. Return the best candidate

Proposed Fix

Port the GUI's selectBestDate logic to Python's _extract_timeframe_from_qualifiers() method:

@staticmethod
def _select_best_date(snaks: list, prefer_latest: bool = False) -> WikidataDate | None:
    """Select best date from multiple qualifier values.
    
    Strategy: prefer most precise date, then earliest for start dates
    or latest for end dates.
    """
    parsed_dates = []
    for snak in snaks:
        if "datavalue" in snak and "value" in snak["datavalue"]:
            value = snak["datavalue"]["value"]
            date = WikidataDate.from_wikidata_time(value["time"], value["precision"])
            if date:
                parsed_dates.append(date)
    
    if not parsed_dates:
        return None
    if len(parsed_dates) == 1:
        return parsed_dates[0]
    
    # Sort by precision (descending), then by date
    def sort_key(d):
        python_date = d.to_python_date()
        date_value = python_date.toordinal() if python_date else 0
        if prefer_latest:
            date_value = -date_value
        return (-d.precision, date_value)
    
    parsed_dates.sort(key=sort_key)
    return parsed_dates[0]

Related

  • Fixes part of the issue identified in #73
  • GUI fix: poliloom-gui/src/lib/wikidata/qualifierParser.ts

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with poliloom/models/politician.py:893-960, especially _extract_timeframe_from_qualifiers(), and compare the existing selection behavior with poliloom-gui/src/lib/wikidata/qualifierParser.ts and selectBestDate(). Done means timeframe comparison, storage, deduplication, and display consistently choose dates by precision and the stated start/end tie-breaking rules.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.