opensanctions / opensanctions/poliloom
API uses naive [0] index for multiple P580/P582 qualifiers in timeframe comparison
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Problem
When comparing position timeframes during import/enrichment, the API extracts only the first qualifier value ([0]) for P580 (start date) and P582 (end date). This is problematic because ~389 positions have multiple P580 values and similar numbers have multiple P582 values.
Affected Code
poliloom/models/politician.py:934-960:
@staticmethod
def _extract_timeframe_from_qualifiers(qualifiers_json: dict | None):
if "P580" in qualifiers_json:
start_data = qualifiers_json["P580"][0]["datavalue"]["value"] # <-- [0] only
start_date = WikidataDate.from_wikidata_time(...)
if "P582" in qualifiers_json:
end_data = qualifiers_json["P582"][0]["datavalue"]["value"] # <-- [0] only
This method is used by:
Property._compare_to()- comparing properties during importProperty.should_store()- deciding whether to store new propertiesProperty.soft_delete_matching_extracted()- deduplication during import
Additionally, Property.format_timeframe() (lines 893-931) also uses [0] for display purposes.
Impact
Incorrect timeframe extraction can cause:
- False matches - positions incorrectly identified as duplicates
- Missed matches - overlapping positions not detected
- Inconsistent display - arbitrary date shown instead of most representative
GUI Solution (for reference)
The GUI solved this in poliloom-gui/src/lib/wikidata/qualifierParser.ts with selectBestDate():
- Parse all date values from the qualifier array
- Sort by precision (most precise first: day > month > year)
- For equal precision: earliest for P580, latest for P582
- Return the best candidate
Proposed Fix
Port the GUI's selectBestDate logic to Python's _extract_timeframe_from_qualifiers() method:
@staticmethod
def _select_best_date(snaks: list, prefer_latest: bool = False) -> WikidataDate | None:
"""Select best date from multiple qualifier values.
Strategy: prefer most precise date, then earliest for start dates
or latest for end dates.
"""
parsed_dates = []
for snak in snaks:
if "datavalue" in snak and "value" in snak["datavalue"]:
value = snak["datavalue"]["value"]
date = WikidataDate.from_wikidata_time(value["time"], value["precision"])
if date:
parsed_dates.append(date)
if not parsed_dates:
return None
if len(parsed_dates) == 1:
return parsed_dates[0]
# Sort by precision (descending), then by date
def sort_key(d):
python_date = d.to_python_date()
date_value = python_date.toordinal() if python_date else 0
if prefer_latest:
date_value = -date_value
return (-d.precision, date_value)
parsed_dates.sort(key=sort_key)
return parsed_dates[0]
Related
- Fixes part of the issue identified in #73
- GUI fix:
poliloom-gui/src/lib/wikidata/qualifierParser.ts
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with poliloom/models/politician.py:893-960, especially _extract_timeframe_from_qualifiers(), and compare the existing selection behavior with poliloom-gui/src/lib/wikidata/qualifierParser.ts and selectBestDate(). Done means timeframe comparison, storage, deduplication, and display consistently choose dates by precision and the stated start/end tie-breaking rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100