koala73 / koala73/worldmonitor

Track import-HHI Comtrade 429 coverage misses

Open
#3,979 0 comments 0 reactions 0 assignees View on GitHub
bug P1
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Context

PR #3978 fixes the deterministic import-HHI coverage bugs: shared non-standard reporter overrides, smaller Comtrade payload shape, strict seed freshness, and a realistic publish floor. One operational risk remains: correct-code reporters can still miss when Comtrade returns 429s during the monthly seed.

Review evidence: a 106-reporter probe on the two production Comtrade keys hit 46 HTTP 429 responses. Payload shaping reduces oversized responses and timeout pressure, but it does not increase Comtrade key budget. The current mitigation is the seeder retry/backoff plus checkpoint/resume accumulation.

## Risk

AE/RU/NO/CH have correct reporter codes and should recover once the seeder completes enough successful fetches. If they remain unmonitored after #3978 deploys and a couple of cron/force-refresh cycles run, the likely cause is still Comtrade rate limiting rather than reporter-code drift.

## Acceptance criteria

- After #3978 deployment, verify `resilience:recovery:import-hhi:v1` and live resilience scores for AE/RU/NO/CH after force-refresh or two monthly cron attempts.
- If those reporters remain absent and seed logs show 429s, increase Comtrade key budget and/or widen request spacing before changing scoring logic.
- Document the operational runbook decision: more `COMTRADE_API_KEYS`, higher `PER_KEY_DELAY_MS`, lower concurrency, or reporter cohort splitting.

## Non-goals

- Do not block #3978 on this follow-up.
- Do not change importConcentration scoring unless the seeded data still fails after the rate-limit path is addressed.

Contributor guide

Open the contributing guide

Research direction

After PR #3978 is deployed, inspect resilience:recovery:import-hhi:v1 and the live resilience scores for AE, RU, NO, and CH after a force-refresh or two monthly cron attempts. Check seed logs for 429s; done means deciding and documenting the operational runbook change, such as COMTRADE_API_KEYS, PER_KEY_DELAY_MS, lower concurrency, or reporter cohort splitting, without changing scoring logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data-engineering, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.