opensanctions / opensanctions/opensanctions
[pg_parliament] Assertion schema_entities failed for Person: 0 is not >= threshold 80
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 810
- Forks
- 187
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 133
Description
The PNG Parliament rebuilt their website (old CodeIgniter-ish site → WordPress + Elementor) sometime in the week before 2026-07-28, and the MP data didn't come along. The crawler emits 0 Persons and just the one hardcoded MP Position, so all three assertions fail. Failing since the 2026-07-28 run; last good run was 20260721042801-lyi (104 Persons, 23 Positions).
No warnings in issues.log, which is the tell — the link XPath matches nothing, so the loop body never executes at all.
What broke
crawl() walks .//a[contains(@href,'parliament/bio/view/')] on the homepage and scrapes each https://www.parliament.gov.pg/index.php/parliament/bio/view/<slug> profile page. Those bio URLs are gone: https://www.parliament.gov.pg/index.php/parliament/bio/view now 404s and redirects to https://parliament.gov.pg/. No bio links left on the homepage.
The new site
/members/ and /governors/ are client-side widgets (<div class="parliament-pg-root" data-view="members" data-wpurl="parliament-pg/v1/get-members">) backed by a custom WP REST API at https://parliament.gov.pg/wp-json/parliament-pg/v1/. The relevant routes:
| Route | Args | Currently returns |
|---|---|---|
get-members |
page, year, categoryId, statusId, sortBy |
{"members":[]} |
get-governors |
page, year, categoryId, statusId, sortBy |
{"governors":[]} |
get-districts |
same | all 89, {"id":1,"name":"Abau","province_id":1,...} |
get-provinces |
same | all 22, {"id":1,"name":"Central","region_code":null,...} |
(GET /wp-json/parliament-pg/v1 gives the full index — there are also bills/hansards/meetings/articles routes.)
So: the member endpoints are empty. Empty with page=1, with sortBy=asc, with blank categoryId/statusId. I also rendered both pages through a real browser to rule out a params mistake — the A–Z filter buttons come back disabled and the card container is empty. The site renders an empty list because the list is empty.
Not a dead site though — districts, provinces and news are all populated. They're mid-migration and haven't loaded the MP records yet. No member post type in wp-sitemap.xml, and no other MP listing anywhere on the new site. demo.parliament.gov.pg is linked all over the markup but doesn't resolve externally.
When it comes back
Rewrite onto the JSON API rather than patching — context.fetch_json, no browser rendering needed (I only used Zyte for the probing above because direct access was slow from here, nothing is blocked). Things to keep in mind then:
- Two endpoints, so the Open-vs-Provincial split comes from which one the record was in, replacing the
slug.startswith("governor-")heuristic. GOVERNOR_POSITIONSQIDs are still good but keyed on the old site's electorate labels; the API's province names differ ("Chimbu (Simbu)"vs"Chimbu Provincial"), so they'll need re-keying, ideally onprovince_id.- Person IDs currently key off the bio URL, so a full re-key and churn delta is unavoidable — worth picking something stable while we're at it.
- 111 seats: 89 Open + 22 Provincial. Current assertion floor of 80 is about right.
🤖 Generated with Claude Code
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the pg_parliament crawl() logic and the schema_entities assertions, then inspect the documented WordPress REST routes when member data is available. Replace the obsolete bio-link traversal with API pagination and account for the Open-versus-Provincial split, stable identifiers, and province mapping. Done means the crawler restores at least 80 Persons and the expected Position assertions pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, wordpress
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100