opensanctions / opensanctions/poliloom
Improve source language detection with an LLM call
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Context
Non-Wikipedia sources (user-submitted URLs like government portals) currently get their language from the <html lang> attribute during archiving (_detect_source_language in poliloom/archiving.py, added in ee4077e). This is brittle: many pages omit the attribute, declare it wrongly, or serve mixed-language content.
Language matters because review serving is language-scoped: a property is only visible to a user if one of its sources has a SourceLanguage row in the user's selected languages. Sources where detection fails fall back to "unknown language = visible to everyone", which dilutes the per-language review model.
Proposal
Replace (or augment) the <html lang> heuristic with an LLM call during source processing:
- Feed the extracted page text (truncated) to the model and ask for the content's language.
- Constrain the output to the current user's language filters — the user who submitted the source (or triggered the enrichment) already told us which languages they review; the page is almost certainly in one of them. Pass their language QIDs as the allowed set (structured output / enum), plus an explicit "other/unknown" option so the model isn't forced into a wrong bucket.
- Map the result to a
Languageentity and link viaSourceLanguage, as today. - Keep "unknown" as a real outcome: no
SourceLanguagerows, preserving the designed fallback instead of guessing.
Open questions
- Which model/reasoning effort (extraction currently uses OPENAI_MODEL with configurable effort)?
- Where exactly in
process_sourcethis belongs relative to archiving and extraction. - Whether Wikipedia sources should ever fall back to this when project relations are missing.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in poliloom/archiving.py by reading _detect_source_language and trace where process_source performs archiving and extraction. Review the existing OPENAI_MODEL configuration and the Language and SourceLanguage relationships before deciding where detection belongs. Done means the detected language is constrained to the submitter’s allowed QIDs, maps to a Language entity, and preserves unknown as no SourceLanguage rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, internationalization
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100