koala73 / koala73/worldmonitor
Track import-HHI Comtrade 429 coverage misses
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Context
PR #3978 fixes the deterministic import-HHI coverage bugs: shared non-standard reporter overrides, smaller Comtrade payload shape, strict seed freshness, and a realistic publish floor. One operational risk remains: correct-code reporters can still miss when Comtrade returns 429s during the monthly seed.
Review evidence: a 106-reporter probe on the two production Comtrade keys hit 46 HTTP 429 responses. Payload shaping reduces oversized responses and timeout pressure, but it does not increase Comtrade key budget. The current mitigation is the seeder retry/backoff plus checkpoint/resume accumulation.
## Risk
AE/RU/NO/CH have correct reporter codes and should recover once the seeder completes enough successful fetches. If they remain unmonitored after #3978 deploys and a couple of cron/force-refresh cycles run, the likely cause is still Comtrade rate limiting rather than reporter-code drift.
## Acceptance criteria
- After #3978 deployment, verify `resilience:recovery:import-hhi:v1` and live resilience scores for AE/RU/NO/CH after force-refresh or two monthly cron attempts.
- If those reporters remain absent and seed logs show 429s, increase Comtrade key budget and/or widen request spacing before changing scoring logic.
- Document the operational runbook decision: more `COMTRADE_API_KEYS`, higher `PER_KEY_DELAY_MS`, lower concurrency, or reporter cohort splitting.
## Non-goals
- Do not block #3978 on this follow-up.
- Do not change importConcentration scoring unless the seeded data still fails after the rate-limit path is addressed.
Contributor guide
Research direction
After PR #3978 is deployed, inspect resilience:recovery:import-hhi:v1 and the live resilience scores for AE, RU, NO, and CH after a force-refresh or two monthly cron attempts. Check seed logs for 429s; done means deciding and documenting the operational runbook change, such as COMTRADE_API_KEYS, PER_KEY_DELAY_MS, lower concurrency, or reporter cohort splitting, without changing scoring logic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data-engineering, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100