GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Migrate SDK ontology pipeline to bigquery_ontology package

Open
#38 14 comments 0 reactions 1 assignee Claimed by @haiyuan-eng-google View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

## Context

The SDK's V5 ontology pipeline (`ontology_models.py`, `ontology_property_graph.py`, `ttl_importer.py`, `ontology_schema_compiler.py`) duplicates functionality now available in the upstream `bigquery_ontology` package: separated ontology/binding models, DDL compiler, OWL importer, scaffold generator, CLI, and ~5K lines of tests.

Migrating the SDK to use `bigquery_ontology` as the foundation would reduce maintenance surface and align with the production ontology contract. However, this is a **runtime contract migration**, not a module swap — the SDK's `GraphSpec` (combined schema + binding) is consumed across extraction, hydration, materialization, schema filtering, DDL compilation, and lineage detection.

## Current state (April 14, 2026)

**SDK V5 pipeline** (haiyuan-eng-google fork, merged):
- `GraphSpec` with embedded `BindingSpec` per entity/relationship
- `ttl_importer.py` — OWL import emitting combined GraphSpec YAML
- `ontology_property_graph.py` — DDL compiler
- `ontology_schema_compiler.py` — AI.GENERATE prompt + output_schema
- `ontology_materializer.py` — batch_load with schema filtering
- `ontology_graph.py` — structured extractor registry + lineage detection
- `BindingSpec.from_session_column` / `to_session_column` for cross-session lineage

**Upstream `bigquery_ontology` package** (782543a):
- Separated `Ontology` + `Binding` models (ontology is backend-neutral)
- `graph_ddl_compiler.py` — DDL compiler (flat ontologies only, v0)
- `owl_importer.py` — OWL import emitting `*.ontology.yaml` (bindings separate)
- `scaffold.py` — generates CREATE TABLE DDL + binding stubs
- `cli.py` — `gm validate`, `gm compile`, `gm import-owl`, `gm scaffold`
- No AI.GENERATE schema compilation
- No `from_session_column` / `to_session_column` in binding model
- Inheritance validated but rejected at compile time (v0)

## Key architectural decision: lineage strategy

The SDK's V5 temporal lineage depends on `from_session_column` / `to_session_column` on `BindingSpec`, which affects:
- `_validate_graph_spec()` — session column validation rules
- `compile_edge_table_clause()` — SOURCE/DESTINATION KEY mapping
- `_route_edge()` — session ownership for delete-scoped idempotency

These fields do not exist in the upstream `binding_models.Binding`. The decision:

**Option A: Upstream the binding extension.**
- Add `from_session_column` / `to_session_column` to upstream `RelationshipBinding`
- Update upstream DDL compiler to use session column overrides
- The SDK compatibility layer becomes transitional scaffolding

**Option B: Permanent SDK-side lineage layer.**
- Keep lineage-specific DDL compilation and materialization in the SDK
- The compatibility layer becomes a permanent runtime boundary
- Upstream package stays focused on the logical ontology contract

This decision should be made before writing migration code because it determines the adapter layer's lifetime and scope.

## Proposed migration plan

### Step 0: ADR — decide lineage strategy
Decide Option A vs Option B above. This gates the adapter design.

### Step 1: Define the adapter contract
Design a minimal `RuntimeSpec` interface that the SDK's extraction, hydration, materialization, schema filtering, and DDL/GQL generation can consume. This interface must be implementable by either:
- The current `GraphSpec` (backward compat)
- `Ontology` + `Binding` from the upstream package

### Step 2: Migrate behind the adapter (in order)
1. **Validation/loading** — `load_graph_spec` delegates to upstream `ontology_loader` + `binding_loader`
2. **OWL import** — `ttl_importer.py` delegates to upstream `owl_importer.py`
3. **DDL compilation** — `ontology_property_graph.py` delegates to upstream `graph_ddl_compiler.py`
4. **Runtime extraction/materialization** — last, because it touches the most call sites

### Step 3: Success criteria
- Existing V5 demo notebook runs unchanged on the new contract
- All existing V5 tests pass (1452+)
- No loss of lineage behavior (cross-session edges, lineage GQL)
- `gm` CLI commands work alongside SDK entry points

## What this issue is for

Discussion on:
1. Is Option A or Option B the right lineage strategy?
2. Is the migration order above correct, or should DDL move before OWL import?
3. Are there upstream changes needed in `bigquery_ontology` beyond lineage (e.g., AI.GENERATE schema support)?
4. Should the adapter layer live in the SDK or in a shared package?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.