[PRD]: Data Quality Enforcement & AI Rule Generation (DQWatch Feature)

Problem Statement

Ontos lets stewards author rich data-quality checks on Data Contracts (dimension, type, severity, SQL/comparators) at contract, schema-object, and property levels — but nothing executes them. The current data_quality_checks job ignores authored DataQualityCheckDb rows entirely; it infers 5 hardcoded checks (required/unique/range/length/pattern) from schema-property flags and runs them via native Spark SQL. So authoring is theater: a steward can define a type='sql' accuracy check with a custom query, and it will never run.

Beyond that gap, stewards have no fast way to create good checks (today only DQX profiling suggests any), no way to set how often checks run per contract or dataset, and no consolidated surface to review what a quality run found.

A separate internal app, DQWatch, already solves all of this — authored rules executed via DQX, LLM/RAG-assisted rule generation from a reusable rule library, per-entity scheduling with inheritance, and a results-review dashboard. We want those capabilities inside Ontos, reusing Ontos's existing contract model, suggestion queue, jobs infrastructure, and Asset Review board rather than standing up a parallel system.

Solution

Deliver end-to-end data-quality enforcement and AI-assisted authoring on top of the existing Data Contract model:

1. Enforcement: Authored ODCS-shaped checks become the source of truth and actually execute, through a pluggable executor abstraction whose first (and, for now, only) implementation translates checks into DQX and runs apply_checks_by_metadata. This replaces the hardcoded native job.
2. AI rule generation: A pluggable suggester engine framework (extracted as shared infrastructure) gains a RAG-based LLM engine: embed the target table's schema + profiling stats, retrieve similar rule templates from a library seeded with DQWatch's published rules, and use an LLM judge to map templates onto the target. Suggestions flow into the existing SuggestedQualityCheckDb review queue alongside DQX profiling suggestions.
3. Scheduling: Stewards set a check interval on a Data Contract; datasets inherit it and can override per dataset. A single dispatcher job reads schedules from the DB and runs what's due.
4. Review: A failing run (error-severity failure or score-threshold breach) creates one Asset Review item per run per contract, auto-assigned to the contract owner for triage.

The suggester framework is deliberately generic so the Ontology Concept/Term Mapping feature (#485) can later ride the same base.

User Stories

1. As a data steward, I want the checks I authored on a contract to actually run against the physical tables, so that quality enforcement reflects my design intent instead of generic heuristics.
2. As a data steward, I want to author a type='sql' check with a custom query and have it executed, so that I can express quality rules that property flags can't capture.
3. As a data steward, I want to author comparator-based checks (must_be_gt, must_be_between, etc.) and have them enforced, so that numeric/range expectations are validated on real data.
4. As a data steward, I want checks defined at the property (column) level and at the schema-object (table) level to both run, so that I can validate at the granularity that fits each rule.
5. As a data steward, I want a run to record which checks passed/failed with counts, so that I can see the quality state of a contract over time.
6. As a data steward, I want dimension-level quality scores (completeness, accuracy, etc.) rolled up from a run, so that I can track quality by dimension on the entity.
7. As a data steward, I want to drill into the actual rows that failed a check, so that I can diagnose and fix underlying data issues.
8. As a data steward, I want AI to suggest quality checks for a table from its schema and profile, so that I don't start from a blank slate.
9. As a data steward, I want AI-suggested checks to appear in the same review queue as DQX profiling suggestions, so that I have one place to accept/reject/edit proposed checks.
10. As a data steward, I want to see why an AI suggested a check (which template it drew from, its rationale), so that I can trust or dismiss it.
11. As a data steward, I want to accept an AI suggestion and have it become a real authored check, so that accepted suggestions immediately participate in enforcement.
12. As a data steward, I want the suggestion library seeded with proven rules, so that AI suggestions are grounded in real-world quality patterns from day one.
13. As a data engineer, I want to set how often a contract's checks run (e.g. daily), so that enforcement matches the data's refresh cadence.
14. As a data engineer, I want a dataset to inherit its contract's schedule by default, so that I don't configure every dataset individually.
15. As a data engineer, I want to override the schedule on a specific dataset, so that a hot table can be checked more often than the rest of the contract.
16. As a data engineer, I want scheduled runs to execute automatically without manual triggering, so that quality monitoring is continuous.
17. As a data engineer, I want to trigger an on-demand run for a contract, so that I can validate immediately after a fix or backfill.
18. As a data owner/steward, I want a review item created when a run has error-severity failures, so that I'm alerted to problems needing attention.
19. As a data owner/steward, I want low-severity/passing runs to NOT create review items, so that my review inbox stays focused on what matters.
20. As a data owner/steward, I want DQ review items auto-assigned to the contract owner, so that accountability follows contract ownership.
21. As a data owner/steward, I want to triage a run (acknowledge/approve/needs-review/deny), so that I can record my assessment of a run's findings.
22. As a data owner/steward, I want a notification when a DQ review item is assigned to me, so that I act promptly.
23. As a data owner/steward, I want to see a run's failing checks and rows from the review detail view, so that I can triage without leaving the review board.
24. As a security-conscious admin, I want LLM rule generation gated behind the one-time consent dialog, so that users explicitly opt in before data is sent to a model.
25. As a security-conscious admin, I want a per-run cap on LLM calls/tokens, so that AI generation cost is bounded.
26. As a security-conscious admin, I want a run that hits the LLM cap to surface a warning rather than fail, so that partial results are still usable.
27. As an admin, I want to configure the score/severity threshold that triggers a review, so that the review board matches our governance policy.
28. As a platform engineer, I want a single dispatcher job rather than one Databricks job per contract, so that scheduling is operationally simple.
29. As a platform engineer, I want the executor behind an interface, so that a non-DQX engine (native/GE/Soda) can be added later without rewrites.
30. As an ontology engineer, I want the suggester/run framework to be domain-agnostic, so that Concept/Term Mapping (#485) can reuse it instead of duplicating run orchestration, consent, and cost accounting.
31. As a steward, I want property-derived checks (the old heuristics) offered as suggested defaults in the review queue, so that I keep that convenience without it silently overriding authored rules at runtime.

Implementation Decisions

Two symmetric pluggable abstractions on a shared run framework:

• Shared suggester/run framework (extracted): Generalize Term Mapping's run lifecycle (pending→suggesting→suggested→applying→applied→undone/failed), engine dispatch loop, and SuggestionDraft shape into a domain-agnostic base. Consumers provide: target adapter(s), suggestion-persistence target, and apply target. Explicit Suggester and TargetAdapter protocols replace the current duck-typing. Term Mapping is NOT migrated onto the base in this effort — DQ is the first consumer that validates the abstraction; #485 migrates Term Mapping later.
• Consent + cost as shared framework infra (built now): Per-run LLM call/token caps + stats accounting wired into run orchestration; consent gate reuses the existing frontend consent dialog. Both DQ and #485 inherit it. New Settings entries for caps and the review threshold.
• Suggester engines: dqx_profiling (refactor of the existing profiling workflow into the engine model) and llm_rag (new). Both emit SuggestedQualityCheckDb rows (source = dqx / llm); accepted suggestions promote to DataQualityCheckDb.
• Executor engines: Executor protocol + a single DqxExecutor. Adds databricks-labs-dqx>=0.14.0 as a backend dependency (not currently present). Translation is ODCS → DQX native check dicts (function + arguments + user_metadata), then apply_checks_by_metadata. The data_quality_checks workflow is rewritten to read authored DataQualityCheckDb for in-scope contracts and run them via the executor.

RAG rule-template library:

• New RuleTemplate + RuleTemplateEmbedding models. Vector stored as a JSON float array in Postgres; cosine similarity computed in pure Python behind a RuleRetriever protocol (mirrors DQWatch; the protocol is the seam to swap in Databricks Vector Search later). Embeddings produced via a Databricks serving endpoint added to app.yaml resources.
• Seed migration ports DQWatch's published rules into RuleTemplate (mapping DQWatch's slot/predicate model onto ODCS-shaped templates). Background re-embed on template publish.

Scheduling:

• Schema change: add schedule fields to DataContractDb (inherited default) and a nullable schedule override on SchemaObjectDb. Resolution: object.schedule ?? contract.schedule.
• A single dispatcher job (frequent cron via existing JobsManager/CronSchedule) reads schedules from Postgres, computes what's due, and runs only those via DqxExecutor. No per-contract Databricks jobs.

Results data model:

• Reuse the existing Postgres run tables (DataQualityCheckRunDb/DataQualityCheckResultDb) for run/aggregate summaries and roll dimension scores into QualityItemDb (source='dqx').
• Port DQWatch's full UC results layer (results fact, run_check_totals, failed_records_latest/quarantine, metric views) for row-level failures and analytics/Genie surface. Postgres = UI metrics/history; UC = drill-into-failing-rows.

Review board:

• New AssetType value DQ_RUN_RESULT (free-string asset_type, no DB enum migration needed); asset_fqn is a run pointer (e.g. dq-run://{contract_id}/{run_id}).
• Executor post-run hook creates one review request per run per contract only when the run has ≥1 error-severity failure or score < configurable threshold, auto-assigned to the contract owner/steward. Resolution is triage-only (does not mutate check definitions). Notifications via the existing trigger registry / NotificationsManager.

Deep modules to extract (simple, isolation-testable interfaces):

• OdcsToDqxTranslator — authored ODCS check → DQX check dict (pure).
• ScheduleResolver — resolve inherited/overridden schedule + compute "due now" (pure).
• ReviewTrigger — run result → should-create-review decision + payload (pure).
• PostgresCosineRetriever — rank templates by cosine over stored embeddings.
• LlmRagSuggester — orchestrate embed→retrieve→judge→draft (LLM + retriever injected).
• RunFramework base — generic run lifecycle + engine dispatch + consent/cost accounting.

Testing Decisions

What makes a good test here: exercise external behavior through the module's public interface, not internal implementation. Prefer pure functions with fixture inputs and asserted outputs; inject/mock only true external boundaries (LLM client, Spark, DB session). No assertions on private helpers or call counts unless they are the contract (e.g. cost cap enforcement).

Modules to be tested (all four requested, plus the RAG glue):

• OdcsToDqxTranslator — table-driven: each ODCS dimension/type/severity/comparator combination → expected DQX check dict. Highest-value target; enforcement correctness hinges on it. Pure, no I/O.
• ScheduleResolver — fixtures for inherited vs overridden schedules and a fixed "now"; assert the resolved cadence and the due/not-due decision. Pure.
• ReviewTrigger — fixture run results (passing, warning-only, error-severity, threshold-breaching); assert whether a review item is created and its assignee/payload. Pure.
• PostgresCosineRetriever + LlmRagSuggester — retriever ranking over fixture embeddings; suggester glue with a mocked LLM + mocked retriever asserting drafts are shaped for SuggestedQualityCheckDb and consent/cost caps are honored (warning at cap, not failure).

Prior art: Term Mapping engine tests mock the FM client and assert draft output (#483 work); existing backend tests already stub source='dqx' suggestions. Follow those patterns and the repo's Alembic single-head migration conventions.

Out of Scope

• Migrating Term Mapping / Concept Mapping onto the extracted framework (that is #485).
• Additional executor engines beyond DQX (native/Great Expectations/Soda).
• Swapping the Postgres+Python retriever for Databricks Vector Search (the protocol seam exists; the swap is a later optimization once the corpus grows).
• Per-check scheduling (only contract + per-dataset override are in scope).
• Fine-grained per-failed-check review items (one item per run per contract only).
• Review resolution feeding back into rule edits (triage-only for now).
• Data-driven (event-triggered) runs; only scheduled + on-demand.
• Per-engine confidence calibration UI and embedding-based concept engines (those live under #485).

Further Notes

• Reference implementation: DQWatch (internally "dqlake"). Port targets: rule_suggest/{retriever,embeddings,refresh}.py + models/rule_embedding.py (RAG), materialiser/{dqx_translator,results_ddl,metric_view,derived}.py (DQX + UC results). Note DQWatch emits DQX-native dicts (no ODCS layer of its own); DQX supports ODCS, so Ontos's translation is ODCS→DQX.
• Delivery: layer-by-layer — (1) shared framework + consent/cost, (2) RAG library + LLM engine, (3) DQX executor + UC results, (4) schedule inheritance + dispatcher, (5) review wiring. This is a multi-PR epic; spawn implementation sub-issues from it.
• Scope honesty: the chosen options (full UC results layer + layer-by-layer) are the heaviest on both axes; nothing works end-to-end until the executor layer lands. If early de-risking becomes a priority, consider pulling a thin Layer-3 tracer bullet (one contract executing end-to-end) forward.
• Related: #485 (Term Mapping LLM-judge engine — shares the framework), #469 (Term Mapping PRD), #172 (Asset Review completion & profiling).

Start with the existing data_quality_checks workflow and the DataQualityCheckDb, DataQualityCheckRunDb, and DataQualityCheckResultDb models mentioned in the proposal. Review the listed executor, suggester, scheduling, retrieval, and review-trigger interfaces and their testing decisions before splitting the work. Done means authored checks execute, suggestions and schedules work, results are reviewable, and the specified external behaviors have tests.