koala73 / koala73/worldmonitor

fuel-prices: restore New Zealand (MBIE) — source moved behind Incapsula JS bot-wall

Open
#4,010 0 comments 0 reactions 0 assignees View on GitHub
bug P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Problem
`scripts/seed-fuel-prices.mjs` `fetchNewZealand()` can no longer retrieve the MBIE weekly fuel CSV. Around **2026-05-20**, `mbie.govt.nz` moved its **entire domain** (apex, the weekly-fuel-price-monitoring data page, and the `weekly-table.csv` asset) behind an **Incapsula (Imperva) JS bot-challenge**. The URL now returns:

- HTTP `200`, `content-type: text/html`, ~212 bytes
- body is an `_Incapsula_Resource` challenge stub, not CSV

Verified blocked from **all** of: a residential IP, a datacenter IP, and the Decodo residential proxy. `data.govt.nz` and `figure.nz` mirrors are also Incapsula-walled. A plain `fetch`/`curl` cannot pass a JS challenge from any IP.

## Interim mitigation (shipped in PR #4007)
`'New Zealand'` added to `TOLERATED_FAILURES` so the dead source no longer rejects the whole validated fuel-prices publish (same precedent as Brazil). `fetchNewZealand()` still runs every cycle and NZ is carried automatically the moment the source returns CSV again — so **if this is fixed, no code change to the validator is needed.**

## To actually restore NZ (pick one)
1. **Headless / challenge-solver fetch** — integrate FlareSolverr / browserless / a Zyte/ScrapingBee unblocker for `fetchNewZealand()` (reusable for any future JS-walled source). New infra + dependency + cost.
2. **Alternative source** — find a non-walled host that publishes the same MBIE weekly board prices (e.g. an official API, a data.govt.nz CKAN resource that serves the raw CSV without the wall, or another aggregator).
3. **Accept NZ as permanently best-effort** like Brazil and close this (data stays absent).

## Acceptance
- `fetchNewZealand()` returns NZ gasoline + diesel in NZD/L again, and
- once NZ is reliably present, optionally remove it from `TOLERATED_FAILURES` to re-tighten the gate.

Context: WM 2026-06-01 DEGRADED triage. See `scripts/seed-fuel-prices.mjs` `TOLERATED_FAILURES` comment + `fetchNewZealand()` (~line 499).

Contributor guide

Open the contributing guide

Research direction

Start with scripts/seed-fuel-prices.mjs, especially TOLERATED_FAILURES and fetchNewZealand() around line 499, and verify the current responses from the listed MBIE and mirror URLs. Compare the three restoration options, then confirm that NZ gasoline and diesel return in NZD/L; if reliable, check whether NZ can be removed from TOLERATED_FAILURES.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.