opensanctions / opensanctions/opensanctions

[us_state_terrorist_orgs] name framework migration

Open
#4,828 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

data-quality
Dominant language
Python
Stars
810
Forks
187
Avg merge
1d 7h
Merged PRs (30d)
133

Description

us_state_terrorist_orgs splits one name string into name/former-name/alias with regex and a lookup fallback instead of using the name-cleaning framework (h.review_names / h.apply_reviewed_name_string).

Crawler: datasets/us/state_terrorist_orgs/crawler.py

Ad-hoc logic to retire (~25 lines):

  • NORMAL_CASE_REcrawler.py:6
  • split_clean_name returning (name, former, alias) by splitting on parentheses, with lookup fallback — crawler.py:11-35, applied at 49-54

Migration (sanctions crawler — 3 steps):

  1. Mirror the existing cleaned names into a suggested h.Names() and call h.review_names(..., default_accepted=True). Output unchanged while reviews accumulate; deploy step 3 ASAP afterwards.
  2. Complete the accumulated name reviews for this dataset.
  3. Replace the custom logic with h.apply_reviewed_names / h.apply_reviewed_name_string and delete it. Add any custom alias-marker phrases not covered by rigour.names.name_split_phrases_list() as reject_strings under names.schema_rules in us_state_terrorist_orgs.yml.

Migration guide: zavod/docs/extract/names.md (section "Migrating to the name cleaning helpers").

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with zavod/docs/extract/names.md, then inspect datasets/us/state_terrorist_orgs/crawler.py and us_state_terrorist_orgs.yml. Run the crawler to establish its current output, review the existing name-cleaning helpers, and identify any alias markers needing names.schema_rules reject_strings. Done means the accumulated reviews are complete, output remains unchanged, and the custom name-splitting logic is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.