opensanctions / opensanctions/opensanctions

[in_mha_banned] name framework migration

Open
#4,822 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

data-quality
Dominant language
Python
Stars
810
Forks
187
Avg merge
1d 7h
Merged PRs (30d)
133

Description

in_mha_banned extracts aliases and acronyms from combined name strings with hand-rolled regex and keyword sets instead of the name-cleaning framework (h.review_names / h.apply_reviewed_name_string).

Crawler: datasets/in/mha_banned/crawler.py

Ad-hoc logic to retire (~45 lines):

  • REGEX_ACRONYM_PARENS pulling acronym-in-parens into aliascrawler.py:15
  • COMPLEX_TERMS keyword set gating a lookup — crawler.py:18-30
  • splitting names_string on ;@ for aliases + parse_namescrawler.py:45-58, 178-184

Migration (sanctions crawler — 3 steps):

  1. Mirror the existing cleaned names into a suggested h.Names() and call h.review_names(..., default_accepted=True). Output unchanged while reviews accumulate; deploy step 3 ASAP afterwards.
  2. Complete the accumulated name reviews for this dataset.
  3. Replace the custom logic with h.apply_reviewed_names / h.apply_reviewed_name_string and delete it. Add any custom alias-marker phrases not covered by rigour.names.name_split_phrases_list() as reject_strings under names.schema_rules in in_mha_banned.yml.

Migration guide: zavod/docs/extract/names.md (section "Migrating to the name cleaning helpers").

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read zavod/docs/extract/names.md, especially the migration section, then inspect datasets/in/mha_banned/crawler.py and datasets/in/mha_banned/in_mha_banned.yml. Trace the existing name handling and review workflow before migrating the dataset and completing its accumulated reviews. Done means the custom extraction logic is removed, schema reject strings cover any missing phrases, and output remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.