databrickslabs / databrickslabs/ontos

[Feature]: Compliance: SCHEMA_DRIFT check function

Open
#129 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/feature
Dominant language
Python
Stars
212
Forks
71
Avg merge
4d 10h
Merged PRs (30d)
43

Description

Parent PRD

#125

What to build

Implement the SCHEMA_DRIFT(physical_name) DSL check function end-to-end:

  • Uses EvaluationContext.workspace_client to run DESCRIBE TABLE on the physical UC table
  • Compares UC columns/types against contract schema properties from the enriched entity dict (requires schema facet)
  • Detects: missing columns (in contract but not in table), extra columns (in table but not in contract), type-changed columns
  • Returns CheckResult with pass/fail, human-readable message, and details_json containing the drift specifics
  • Reuses logic from the existing data_contract_validation.py check_schema_drift function
  • Register in the check function registry
  • Tests with mocked WorkspaceClient covering: no drift, missing columns, extra columns, type changes, empty schema

Acceptance criteria

  • SCHEMA_DRIFT function registered in check function registry
  • Compares contract schema properties against UC DESCRIBE TABLE output
  • Correctly identifies missing, extra, and type-changed columns
  • Returns structured details_json with drift specifics
  • Tests with mocked WorkspaceClient cover all drift scenarios

Blocked by

  • Blocked by #127 (EvaluationContext + check function registry)
  • Blocked by #128 (facet-based entity enrichment for schema facet)

User stories addressed

  • User story 9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This work depends on #127 and #128; start with the EvaluationContext and check function registry entry points, then review the existing check_schema_drift function in data_contract_validation.py. Use mocked WorkspaceClient tests for no drift, missing columns, extra columns, type changes, and an empty schema; done means the registered SCHEMA_DRIFT function returns structured drift details and correct pass/fail results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.