epam / epam/statgpt-backend

Support the "Indicator-per-Dataflow" SDMX data model

Open
#489 0 comments 0 reactions 1 assignee Claimed by @Fedir-Yatsenko View on GitHub
enhancement
Dominant language
Python
Stars
26
Forks
1
Avg merge
1d 17h
Merged PRs (30d)
25

Description

### StatGPT Backend version

0.0.0

### What is the problem this feature will solve?

StatGPT today assumes the **conventional** SDMX shape: a dataset is a single dataflow that
holds many indicators, and the specific indicator is chosen from a dimension inside that
dataflow (e.g. an indicator/measure dimension). The user picks a dataflow, then narrows
down to a series.

A large class of providers does not follow this shape. They publish **one narrow table —
effectively one indicator — per dataflow**, resulting in thousands of dataflows, each with
its own structure. There is no broad dataflow to select and filter; the indicator is
determined by *which* of the thousands of dataflows is chosen.

Because of this, StatGPT cannot currently index, select, or query these providers, and
they are marked unsupported. The following providers are affected:

| Provider (unlocked) | Scope | Approx. dataflows |
|---------------------|-------|-------------------|
| Eurostat (ESTAT) | EU | ~3,900 |
| OECD | International | ~1,500 |
| ISTAT | Italy | ~1,500 |
| ABS | Australia | ~1,200 |
| ILO | International | ~1,200 |

Countries whose national or regional source becomes usable once this is solved:
**Italy** (ISTAT), **Australia** (ABS), and the **EU / EEA** (Eurostat).

Supporting this model extends StatGPT's coverage of official statistics to these providers
and countries.

### What is the proposed feature or solution?

StatGPT should be able to ingest, index, present, and query providers that expose thousands
of single-indicator dataflows, with the same answer quality and user experience it offers
for conventional providers. At the business level this means:

- Treat a provider's large set of narrow dataflows as a browsable, searchable **catalog of
indicators**, rather than requiring one big dataflow with an indicator dimension.
- Let a user find and retrieve a **specific indicator** even when that indicator is an
entire dataflow of its own.
- Handle the **scale** of these providers (thousands, and for the largest tens of
thousands, of dataflows) in discovery, indexing, and retrieval without degrading
relevance or performance.
- Achieve all of the above **without asking the data provider to restructure** their data.

**Success criteria:**

- At least one indicator-per-dataflow provider (target: **Eurostat**) can be added as a
data source and answers user questions end-to-end.
- Users can discover and retrieve a specific indicator from such a provider as easily as
from a conventional provider.
- Discovery/indexing completes and stays performant at the scale of the largest provider
listed above.
- The provider is onboarded without any structural changes on the provider side.

**Out of scope:**

- Any change to conventional providers already supported (they must keep working as-is).
- Connectivity, authentication, and performance requirements defined elsewhere in the SDMX
compatibility documentation — those remain unchanged.
- Asking data providers to change their published data model.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.