Support Grade B datasets — registry-level onboarding with dataset-level discovery

## StatGPT Backend version

latest

## What is the problem this feature will solve?

Today a dataset is only usable in StatGPT after full onboarding: an administrator configures it, and it is indexed at the series level. This yields excellent search quality, but the onboarding effort per dataset is high, so coverage grows slowly. Everything outside onboarded coverage is invisible — when a user asks for data that has not been onboarded, the agent cannot recognize that a relevant official dataset exists, and cannot point the user to it.

We want a lighter onboarding grade that trades search quality for coverage. Let's call the current full onboarding **Grade A**, and the new registry-level onboarding **Grade B**.

Grade B makes datasets from an SDMX registry discoverable at a fraction of the onboarding cost: an administrator registers a whole registry rather than individual datasets, and StatGPT indexes only dataset-level metadata. In return, the agent can refer the user to relevant datasets it could not previously see, and — where the dataset's structure permits — build a query against it on demand.

## What is the proposed feature or solution?

Introduce **Grade B** onboarding: registry-level onboarding with dataset-level (dataflow-level) metadata indexing and dataset-first querying.

**Onboarding unit.** The administrator registers an SDMX registry, not individual datasets. A new entity stores the registry configuration. The registry is reached through the existing StatGPT SDMX Proxy — no new connection mechanism.

**Indexing.** Only dataset metadata is indexed, never the data itself. For each dataflow in the registry, the admin backend extracts metadata from the dataflow, its dataset-level attributes, and its DSD, and maps it into a shared discovery record (agency, reference-area coverage, id/name/description, source URL, time / frequency / indicator / unit coverage). Codelists are not indexed verbatim — they are summarized into short coverage metadata. Records are stored in a dedicated Generic RAG channel, one per StatGPT channel, shared with other discovery-grade records. Reindexing is administrator-triggered; scheduling is a follow-up.

**Search & referral.** A discovery search tool queries the Generic RAG channel — pre-filtered by reference area and agency, then hybrid semantic + keyword over the remaining metadata. The agent presents the most relevant candidate datasets with links to their official source, and asks the user whether they want to query one of them.

**Querying.** If the user confirms a dataset, a separate tool builds the query against that registry from the dataset's SDMX structure metadata at request time, using the availability and data endpoints exposed through the proxy. No search indexes are involved — Grade B has no series-level index.

**Not in this scope.** Grade A onboarding is unchanged. Indexing-time record validation is deferred. How the discovery tools are exposed to the agent (a dedicated tool versus a fallback path within existing tools) is still open.

## What alternatives have you considered?

- **Series-level indexing for Grade B datasets** — rejected; it is exactly the cost that makes Grade A hard to scale, and it defeats the purpose of a lighter grade.
- **Reusing StatGPT's own pgvector/Elastic index for discovery records** — rejected in favor of the Generic RAG application, which already provides indexing, hybrid retrieval, and metadata filtering.
- **Per-dataset onboarding with a reduced configuration** — rejected; the point of Grade B is that coverage grows per registry, not per dataset. Registries following the indicator-per-dataflow pattern can contain thousands of dataflows.

## See also

- Grade C: #546