opensanctions / opensanctions/poliloom

Enhancement: Order enrichment by metadata completeness (prioritize sparse politicians)

Open
#123 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

loom
Dominant language
Python
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

Currently, the enrichment process orders politicians for enrichment based on:

  1. Never-enriched first (enriched_at IS NULL)
  2. Oldest enrichment first
  3. Newer politicians first (by QID descending)

While this ensures fair coverage, it doesn't account for metadata completeness. Politicians with sparse data (few properties) should ideally be enriched before those with rich data.

Proposed Solution

Order politicians for enrichment by a metadata completeness score:

ORDER BY
  property_count ASC,           -- Fewest properties first (sparse data)
  wikilink_count DESC,          -- Most Wikipedia links first (rich extractable content)
  wikidata_id_numeric DESC      -- Newer politicians as tiebreaker
Rationale
  • Property count ASC: Target politicians who need data most
  • Wikilink count DESC: Prioritize politicians with rich Wikipedia pages (more links = more content to extract), maximizing extraction yield
  • QID as tiebreaker: Standard ordering for consistency
Implementation Approach

Add scalar subqueries in enrich_politician_from_wikipedia():

# Count non-deleted properties per politician
property_count = (
    select(func.count())
    .where(
        and_(
            Property.politician_id == Politician.id,
            Property.deleted_at.is_(None),
        )
    )
    .correlate(Politician)
    .scalar_subquery()
)

# Count wikipedia links per politician
wikilink_count = (
    select(func.count())
    .where(WikipediaLink.politician_id == Politician.id)
    .correlate(Politician)
    .scalar_subquery()
)

.order_by(
    property_count.asc(),
    wikilink_count.desc(),
    Politician.wikidata_id_numeric.desc(),
)

Current Code Location

poliloom/enrichment.py:890-893 - Current ORDER BY clause

Benefits

  • Systematic gap filling: Focuses on politicians with least known metadata
  • Efficient enrichment: Extract data for entities that need it most AND have content available
  • Better resource utilization: Avoids over-enriching already data-rich politicians

Implementation Notes

  • Should work with existing filters (languages, countries, stateless)
  • Property count only considers non-deleted properties
  • May want to add index on properties(politician_id) WHERE deleted_at IS NULL if performance becomes an issue

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read poliloom/enrichment.py:890-893 and the enrich_politician_from_wikipedia() entry point first. Implement the proposed ordering while preserving the existing language, country, and stateless filters. Done means non-deleted property counts, Wikipedia link counts, and the numeric QID determine the enrichment order as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlalchemy
Domain
backend, databases
Issue type
Feature
Difficulty
2/5
Estimated time
Half a day
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.