koala73 / koala73/worldmonitor
fuel-prices: restore New Zealand (MBIE) — source moved behind Incapsula JS bot-wall
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Problem
`scripts/seed-fuel-prices.mjs` `fetchNewZealand()` can no longer retrieve the MBIE weekly fuel CSV. Around **2026-05-20**, `mbie.govt.nz` moved its **entire domain** (apex, the weekly-fuel-price-monitoring data page, and the `weekly-table.csv` asset) behind an **Incapsula (Imperva) JS bot-challenge**. The URL now returns:
- HTTP `200`, `content-type: text/html`, ~212 bytes
- body is an `_Incapsula_Resource` challenge stub, not CSV
Verified blocked from **all** of: a residential IP, a datacenter IP, and the Decodo residential proxy. `data.govt.nz` and `figure.nz` mirrors are also Incapsula-walled. A plain `fetch`/`curl` cannot pass a JS challenge from any IP.
## Interim mitigation (shipped in PR #4007)
`'New Zealand'` added to `TOLERATED_FAILURES` so the dead source no longer rejects the whole validated fuel-prices publish (same precedent as Brazil). `fetchNewZealand()` still runs every cycle and NZ is carried automatically the moment the source returns CSV again — so **if this is fixed, no code change to the validator is needed.**
## To actually restore NZ (pick one)
1. **Headless / challenge-solver fetch** — integrate FlareSolverr / browserless / a Zyte/ScrapingBee unblocker for `fetchNewZealand()` (reusable for any future JS-walled source). New infra + dependency + cost.
2. **Alternative source** — find a non-walled host that publishes the same MBIE weekly board prices (e.g. an official API, a data.govt.nz CKAN resource that serves the raw CSV without the wall, or another aggregator).
3. **Accept NZ as permanently best-effort** like Brazil and close this (data stays absent).
## Acceptance
- `fetchNewZealand()` returns NZ gasoline + diesel in NZD/L again, and
- once NZ is reliably present, optionally remove it from `TOLERATED_FAILURES` to re-tighten the gate.
Context: WM 2026-06-01 DEGRADED triage. See `scripts/seed-fuel-prices.mjs` `TOLERATED_FAILURES` comment + `fetchNewZealand()` (~line 499).
Contributor guide
Research direction
Start with scripts/seed-fuel-prices.mjs, especially TOLERATED_FAILURES and fetchNewZealand() around line 499, and verify the current responses from the listed MBIE and mirror URLs. Compare the three restoration options, then confirm that NZ gasoline and diesel return in NZD/L; if reliable, check whether NZ can be removed from TOLERATED_FAILURES.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100