koala73 / koala73/worldmonitor

fix(consumer-prices): recover enabled retailer extraction across eight partial markets

Open
#6,182 1 comment 0 reactions 0 assignees View on GitHub
bug P2
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Summary

Priority: **P2**.

All eight consumer-price markets successfully publish their final 12/12 category snapshots, but every market currently reports `COVERAGE_PARTIAL` because enabled retailer extraction is losing pages or rejecting observations. This is a persistent source-quality problem, not a publication outage and not an observability gap: the coverage metadata added under #5945 is correctly exposing it.

The two weakest markets are India at **54.17%** page completion and Saudi Arabia at **61.11%**. Several individual retailers are substantially worse.

## Production evidence

Fresh compact health snapshot at **2026-08-04T19:58:33Z**:

| Market | Completed / attempted pages | Completion | Failed pages | Rejected observations |
|---|---:|---:|---:|---:|
| AE | 37 / 48 | 77.08% | 11 | 5 |
| AU | 23 / 24 | 95.83% | 1 | 5 |
| BR | 22 / 22 | 100% | 0 | 5 |
| GB | 22 / 24 | 91.67% | 2 | 2 |
| IN | 13 / 24 | 54.17% | 11 | 1 |
| SA | 22 / 36 | 61.11% | 14 | 4 |
| SG | 20 / 24 | 83.33% | 4 | 5 |
| US | 24 / 24 | 100% | 0 | 3 |

The largest enabled-retailer gaps are:

| Retailer | Completed / attempted | Completion |
|---|---:|---:|
| JioMart India | 1 / 12 | 8.33% |
| Noon Grocery UAE | 3 / 12 | 25.00% |
| Noon Saudi Arabia | 4 / 12 | 33.33% |
| Carrefour Saudi Arabia | 7 / 12 | 58.33% |
| Cold Storage Singapore | 8 / 12 | 66.67% |

BigBasket India remains 12/12, showing that the India market failure is retailer-specific rather than a market-wide publisher failure. AnaNinja Saudi Arabia is 11/12, likewise isolating most Saudi loss to Carrefour and Noon.

Brazil and the United States complete every requested page but remain partial because their strict validators reject observations. Those rejections must be classified as legitimate source-quality protection versus correctable parse/unit/schema errors; zero failures alone does not prove complete usable coverage.

The scrape, aggregate, and publish stages completed, and the final consumer-price product published 12/12 category snapshots for every market. This issue therefore owns input recovery while preserving truthful partial status until the underlying observations recover.

## Existing issue boundaries

- #5445 owns implausible price deltas, unit/pack-size parse artifacts, and malformed mover/spread output.
- #5811 concerns currently disabled retailers and an external provider proposal.
- #5945 delivered per-market/per-retailer coverage observability and strict-validator reporting.

This issue owns the **enabled retailers currently returning partial page coverage** and the classification of current rejection counts. It must cross-link rather than duplicate provider evaluation or parse-artifact work.

## Scope

1. Reproduce each enabled retailer failure from the production/Railway network and classify it by retailer, page/category, provider route, HTTP/result class, and final rejection reason.
2. Prioritize JioMart India, Noon UAE, Noon Saudi Arabia, Carrefour Saudi Arabia, and Cold Storage Singapore.
3. Determine whether each gap is caused by upstream blocking, provider extraction failure, stale selector/schema assumptions, wrong-domain output, timeout/rate limiting, or a correct strict-validator rejection.
4. Repair the existing adapter/provider route where sustainable. If replacement-provider work is required, coordinate it with #5811 rather than silently introducing an unreviewed dependency.
5. Preserve the strict price, size, currency, domain, and plausibility validators. Do not make health green by accepting malformed observations.
6. Keep per-retailer completion, failure, rejection, and provider diagnostics in seed metadata and health.
7. Bound retries and proxy/provider fallback; do not repeatedly call a known-failing route throughout the same scrape.
8. Preserve the final 12/12 category snapshot contract and last-known-good behavior while sources recover.

## Acceptance criteria

- [ ] Production diagnostics identify the exact failure class for every currently failed page from JioMart IN, Noon AE, Noon SA, Carrefour SA, and Cold Storage SG.
- [ ] JioMart IN, Noon AE, Noon SA, Carrefour SA, and Cold Storage SG each complete at least 10/12 pages, or a separately approved provider/retailer replacement restores equivalent category coverage.
- [ ] India and Saudi Arabia each reach at least 90% market page completion.
- [ ] Every enabled market reaches at least 90% page completion with no enabled retailer below 75%.
- [ ] Every remaining rejected observation has a bounded reason code and is demonstrated to be either a legitimate validator rejection or covered by #5445 as a parse-quality defect.
- [ ] Validators are not weakened and existing implausible-price fixtures remain rejected.
- [ ] All eight markets continue publishing 12/12 final category snapshots.
- [ ] Health and seed metadata continue reporting retailer-level completion, failure, rejection, and provider state.
- [ ] Retry/provider-error volume remains bounded and known failures enter observable cooldown rather than tight retries.
- [ ] Tests cover the diagnosed retailer failures, recovered extraction, validator preservation, partial fallback, and final publication.
- [ ] Two consecutive scheduled scrape → aggregate → publish cycles meet the recovery thresholds before closure.

## Related

- #5445 — parse artifacts and implausible price movements
- #5811 — disabled-retailer recovery and provider proposal
- #5945 — ingestion coverage observability

Contributor guide

Open the contributing guide

Research direction

Start at the scrape, aggregate, and publish stages, then reproduce the enabled-retailer failures through the existing adapter/provider routes and inspect seed metadata and health diagnostics. Prioritize JioMart IN, Noon UAE and Saudi Arabia, Carrefour Saudi Arabia, and Cold Storage Singapore; done means recovered bounded extraction, preserved validators and 12/12 publication, covered tests, and two successful scheduled cycles.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, data-engineering, observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.