epam / epam/statgpt-backend

Grade B: dataset-first querying for a selected dataset

Open
#552 0 comments 0 reactions 2 assignees Claimed by @navalnica View on GitHub
enhancement
Dominant language
Python
Stars
26
Forks
1
Avg merge
1d 17h
Merged PRs (30d)
25

Description

## StatGPT Backend version

latest

## What is the problem this feature will solve?

Part of #545. Depends on #548 and #551.

Discovery search refers users to candidate datasets, and for Grade C that is the end of the flow. Grade B's additional promise is that a referred dataset can be queried on the spot — that is the whole difference between the two grades.

Neither half of that exists today. The referral ends at a source link regardless of grade, and the existing data query pipeline cannot serve these datasets at all: it resolves a request through the series-level index, which Grade B datasets by definition do not have.

## What is the proposed feature or solution?

Add dataset-first querying for a Grade B dataset the user selected from a referral.

**Referral branch.** For a Grade B candidate, the agent offers to build a query instead of ending at the source link; Grade C candidates keep the referral-only behavior. The offer is part of how candidates are presented (#548).

**Carrying the selection.** The user's confirmation arrives on a later turn than the referral, so the selected dataset has to survive across turns and reach the query tool. Existing tools are single-shot and carry state through `ToolMessageState`, so this is new ground and worth settling deliberately rather than improvising.

**Query tool.** A separate tool builds the query at request time from the dataset's SDMX structure: load the structure through the proxy, map the request onto dimensions using their codelists directly — no index and no indicator selection — then resolve against the availability endpoint and fetch from the data endpoint. It reuses the proxy client and the dimension models, and likely the existing query finalization and execution stages, but replaces indicator selection entirely, since that stage is precisely what depends on the series index.

**Best-effort semantics.** Query results are assured at the level of the grade, not per dataset. When a query cannot be built, or returns nothing, the tool falls back to the referral link rather than failing the turn.

**Not in this scope.** Whether a given dataset is queryable at all — the cardinality gate — is determined separately. Grade C querying does not exist. Evaluation of query building is separate work.

## What alternatives have you considered?

- **Querying without asking the user to confirm a dataset** — rejected; dataset-first search is structurally weaker than series-level search, so picking the wrong dataset silently produces a confidently wrong answer. Explicit selection keeps the user in the loop at the point where the risk is highest.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.