opensanctions / opensanctions/poliloom

Use Wikidata P1412 (languages spoken) property for direct language inference

Open
#69 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

loom
Dominant language
Python
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Problem

Currently, we infer which languages a politician might speak by using an indirect path:

  1. Get politician's citizenship(s) (P27)
  2. Look up official languages (P37) of those countries via WikidataRelation
  3. Use those languages to select Wikipedia articles for enrichment

This works but is indirect and potentially incomplete, as:

  • Not all politicians speak the official languages of their citizenship country
  • Politicians may speak languages not official in their citizenship country
  • The inference requires maintaining country→language relations

Solution

Wikidata provides P1412 (languages spoken, written or signed) which directly specifies languages a person can use. When this property is set for a politician, we should use it.

Implementation

The system already automatically imports all relations defined in RelationType enum via extract_all_relations() in wikidata_entity_processor.py:171-209. This method iterates through all RelationType values and extracts those relations.

Steps:

  1. Add P1412 to RelationType enum (models.py:48-58):

    LANGUAGES_SPOKEN = "P1412"  # Languages spoken, written or signed
    
  2. Update language selection logic (models.py:323-389 in get_priority_wikipedia_links):

    • Check if politician's WikidataEntity has P1412 relations (via wikidata_entity.parent_relations)
    • If yes, prefer Wikipedia links in those languages
    • If no, fall back to current citizenship→official language logic
    • Maintain existing popularity-based ordering within selected languages
  3. Re-import entities (optional - for existing data):

    • Run poliloom import-entities to populate P1412 relations for existing politicians
    • New imports will automatically include P1412 relations

Benefits

  • More accurate language selection when P1412 is available
  • Direct property usage instead of inference
  • Maintains backward compatibility (fallback to citizenship logic)
  • Better data quality for enrichment
  • Automatic import - no changes needed to import logic

Related Code

  • wikidata_entity_processor.py:171-209 - extract_all_relations() (auto-imports all RelationType)
  • models.py:48-58 - RelationType enum
  • models.py:323-389 - get_priority_wikipedia_links() method
  • enrichment.py:576-577 - Where priority links are used

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in models.py by reading RelationType and get_priority_wikipedia_links(), then trace extract_all_relations() in wikidata_entity_processor.py to confirm how relations are imported. Done means P1412 languages are preferred when present, citizenship-based language selection remains the fallback, and existing popularity ordering is preserved; enrichment.py shows where the links are consumed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.