Support the "Indicator-per-Dataflow" SDMX data model
- Dominant language
- Python
- Stars
- 26
- Forks
- 1
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 25
Description
### StatGPT Backend version
0.0.0
### What is the problem this feature will solve?
StatGPT today assumes the **conventional** SDMX shape: a dataset is a single dataflow that
holds many indicators, and the specific indicator is chosen from a dimension inside that
dataflow (e.g. an indicator/measure dimension). The user picks a dataflow, then narrows
down to a series.
A large class of providers does not follow this shape. They publish **one narrow table —
effectively one indicator — per dataflow**, resulting in thousands of dataflows, each with
its own structure. There is no broad dataflow to select and filter; the indicator is
determined by *which* of the thousands of dataflows is chosen.
Because of this, StatGPT cannot currently index, select, or query these providers, and
they are marked unsupported. The following providers are affected:
| Provider (unlocked) | Scope | Approx. dataflows |
|---------------------|-------|-------------------|
| Eurostat (ESTAT) | EU | ~3,900 |
| OECD | International | ~1,500 |
| ISTAT | Italy | ~1,500 |
| ABS | Australia | ~1,200 |
| ILO | International | ~1,200 |
Countries whose national or regional source becomes usable once this is solved:
**Italy** (ISTAT), **Australia** (ABS), and the **EU / EEA** (Eurostat).
Supporting this model extends StatGPT's coverage of official statistics to these providers
and countries.
### What is the proposed feature or solution?
StatGPT should be able to ingest, index, present, and query providers that expose thousands
of single-indicator dataflows, with the same answer quality and user experience it offers
for conventional providers. At the business level this means:
- Treat a provider's large set of narrow dataflows as a browsable, searchable **catalog of
indicators**, rather than requiring one big dataflow with an indicator dimension.
- Let a user find and retrieve a **specific indicator** even when that indicator is an
entire dataflow of its own.
- Handle the **scale** of these providers (thousands, and for the largest tens of
thousands, of dataflows) in discovery, indexing, and retrieval without degrading
relevance or performance.
- Achieve all of the above **without asking the data provider to restructure** their data.
**Success criteria:**
- At least one indicator-per-dataflow provider (target: **Eurostat**) can be added as a
data source and answers user questions end-to-end.
- Users can discover and retrieve a specific indicator from such a provider as easily as
from a conventional provider.
- Discovery/indexing completes and stays performant at the scale of the largest provider
listed above.
- The provider is onboarded without any structural changes on the provider side.
**Out of scope:**
- Any change to conventional providers already supported (they must keep working as-is).
- Connectivity, authentication, and performance requirements defined elsewhere in the SDMX
compatibility documentation — those remain unchanged.
- Asking data providers to change their published data model.
Contributor guide
Assessment
This issue has not been assessed yet.