koala73 / koala73/worldmonitor

feat(seeds): OECD SDMX — parked pending concrete consumer

Open
#3,025 1 comment 1 reaction 0 assignees View on GitHub
area: API feature P3
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

## Background

OpenEcon exposes the full OECD SDMX 3.0 API (no key required). WorldMonitor currently has **zero OECD coverage**. OECD publishes governance, labor, tax, education, and health dimensions across 38 OECD members.

## Rescope (2026-04-13): hold for a concrete consumer — do NOT seed yet

Review across issues #3025–#3028 placed OECD lowest priority. Rationale:

- **Coverage ceiling (38 members)** conflicts with WorldMonitor's "world coverage, never subset" principle — any OECD-only data point would leave every non-OECD country showing a blank tile or (worse) force a scorer to gate a dimension on OECD-only inputs.
- **Most indicators duplicate existing seeds**: GDP / CPI / unemployment / fiscal balances are already covered by IMF WEO (expanded further in #3027). Tax revenue %GDP is already seeded via IMF `GGR_G01_GDP_PT`.
- **No current consumer in the codebase**: the resilience dimension registry (`server/worldmonitor/resilience/v1/_indicator-registry.ts`) has no slot waiting on OECD education/R&D/health %GDP inputs; no panel today renders those fields.

This issue stays OPEN as a parking lot for future OECD use-cases that aren't yet scoped. When a concrete consumer emerges, re-evaluate.

## Candidate future use-cases

Track these here rather than close. Each needs a concrete panel or scoring input before seeding:

| OECD dataflow | Future consumer trigger |
|---|---|
| `OECD.SDD.TPS / DSD_PDB@DF_PDB_LV` (labor productivity levels) | Only OECD source with direct substitute; revisit if a "competitiveness" dimension lands in resilience |
| `OECD.STI.PIE` (R&D % GDP) | Revisit if an "innovation capacity" dimension is added to resilience scoring |
| `OECD.SDD.EDSTAT` (education spending % GDP) | Revisit if a social-investment dimension is added |
| `OECD.SDD.TPS / DSD_SHA` (health expenditure) | Revisit if `healthPublicService` expands beyond current WHO/FSI inputs |
| `OECD.ELS.SAE / DSD_HW@DF_AVG_ANN_HRS_WKD` (average hours worked) | Only OECD source; revisit if a labor-market tile lands for OECD-only overlay |
| Governance indicators (product market regulation, rule-of-law index) | V-Dem / WGI already cover this more broadly |

When scoping any of the above: include explicit behavior for non-OECD countries (hide the tile, not blank it), and pair with a world-coverage alternative wherever possible.

## Technical notes (kept for future reference)

- **Base URL:** `https://sdmx.oecd.org/public/rest`
- **Endpoint:** `/data/{agency},{dsd_id}@{dataflow_id},{version}/{filter_key}?dimensionAtObservation=AllDimensions&startPeriod=2019&endPeriod=2025`
- **Accept header:** `application/vnd.sdmx.data+json; version=2.0.0`
- **Country format:** ISO3 (USA, DEU, GBR). Aggregates: `OECD`, `EA19`, `EU27_2020`, `G7`, `G20`
- **Filter key:** dot-separated dimension indices (e.g., `.USA..........`). Position varies per DSD — must introspect the DSD first via `/datastructure/{agency}/{dsd_id}/{version}` with `Accept: application/vnd.sdmx.structure+json; version=2.0.0`
- **Rate limit:** ~60 req/hr per IP. Circuit breaker + exponential backoff (3 attempts: 3s, 6s, 12s with jitter). Timeout 50s per request (OECD is slow).
- **No auth required**

### Sample working URL

```
curl -H "Accept: application/vnd.sdmx.data+json; version=2.0.0" \
"https://sdmx.oecd.org/public/rest/data/OECD.SDD.NAD,DSD_NAMAIN10@DF_TABLE1_EXPENDITURE,1.0/.USA..........?dimensionAtObservation=AllDimensions&startPeriod=2020&endPeriod=2025"
```

### Response parse pattern

```js
const observations = json.data.dataSets[0].observations;
const dims = json.data.structures[0].dimensions.observation;
const timeValues = dims.find(d => d.id === 'TIME_PERIOD').values;
for (const [obsKey, obsVal] of Object.entries(observations)) {
const indices = obsKey.split(':').map(Number);
const timePeriod = timeValues[indices.at(-1)].id;
const value = obsVal[0];
}
```

### Gotchas

- OECD does not reliably populate the `position` field on dimensions — map by ID, not index alone
- DSDs vary per dataflow — do not assume a shared filter-key shape

## Acceptance criteria (for future activation)

When this issue is un-parked:

- [ ] A specific resilience dimension or panel tile must be named as the consumer BEFORE any seeder lands
- [ ] Non-OECD country rendering path is explicit (hidden vs blank) and doesn't degrade existing panels
- [ ] Pair with a world-coverage fallback source where possible

Contributor guide

Open the contributing guide

Research direction

No implementation is requested while this issue is parked; start by reading server/worldmonitor/resilience/v1/_indicator-registry.ts and identifying a concrete panel or resilience consumer. Activation is complete only when that consumer is named, non-OECD rendering is explicit, and a world-coverage fallback is paired where possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.