epam / epam/statgpt-backend

Grade B: validate registry-sourced records before indexing

Open
#558 0 comments 0 reactions 1 assignee Claimed by @Fedir-Yatsenko View on GitHub
enhancement
Dominant language
Python
Stars
26
Forks
1
Avg merge
1d 17h
Merged PRs (30d)
25

Description

## StatGPT Backend version

latest

## What is the problem this feature will solve?

Part of #545. Extends #551, and builds on the shared validation gate introduced in #557.

Grade B records are derived from a registry's own metadata, so their failure modes differ from Grade C's. The registration guardrails in #551 bound a registry by its declared regional coverage and agency, but they act on the registry as a whole, not on each record: a registry that registers legitimately can still expose individual dataflows whose metadata claims coverage beyond that remit, or whose fields carry content unrelated to the dataset.

There is also no collection-side confirmation step here at all — nobody reviews a registry's dataflows before StatGPT reads them.

## What is the proposed feature or solution?

Validate Grade B records against the registry that supplied them, reusing the shared gate.

**Reuse the shared framework.** The validation gate, per-record isolation, and run reporting come from #557. This issue adds the checks specific to registry-sourced records.

**Per-record coverage and agency check.** Validate each dataflow's record against the registry's declared regional coverage and agency. A record claiming coverage beyond what the registry declared is rejected, so the guardrails hold per dataflow and not merely at registration.

**Malformed and misleading metadata.** Apply the same content checks as Grade C to the registry-sourced fields — the dataflow, its dataset-level attributes, and its DSD.

**Cost at scale.** A registry can hold thousands of dataflows, so per-record validation cost is a design constraint rather than a detail. Cheap deterministic checks should run first, with more expensive judgement applied only where they are inconclusive.

**Reporting.** Failures are reported per registry run alongside the per-dataflow read failures already isolated in #551, so one report covers everything that kept a dataflow out of the index.

**Not in this scope.** The registration guardrails themselves (#551), Grade B querying, and measuring how accurately validation classifies records.

## What alternatives have you considered?

- **Relying on the registration guardrails alone** — rejected; they constrain what a registry may register, not what each of its dataflows actually claims.
- **Reusing the Grade C checks from #557 unchanged** — insufficient; Grade C records are self-declared descriptions with no reference to check against, whereas a Grade B record can be checked against its registry's declared attributes. The jurisdiction check is stronger here and should use that.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.