apache / apache/ossie

[databricks] MV→Ossie import aborts on non-equi/filtered join conditions instead of preserving them in custom_extensions

Open
#321 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
267
Avg merge
4d 20h
Merged PRs (30d)
24

Description

## Summary
`convert_metric_view_to_ossie` raises `ConversionError` and aborts the entire import when a Metric View join's `on` clause is not a simple equi-join of `alias.column = alias.column` pairs. Real-world metric views routinely use function-wrapped keys and filter predicates in join conditions, so this makes the importer unusable on them — even though the converter already has a custom_extensions mechanism that could preserve such joins losslessly.

## Current behavior
`_decompose_on()` returns the raw `on` string for any clause containing a non-equi operator, a SQL fragment (e.g. `UPPER(...)`, `COALESCE(...)`), or an extra filter predicate (`... AND col NOT IN (...)`). `_convert_join()` then:

if raw_on is not None:
raise ConversionError("Join '...' uses a non-equi or unsupported join
condition ... Cannot import.")

Example (generic Databricks Unity Catalog metric view):
```
joins:
- name: accounts
source: catalog.schema.account
on: UPPER(source.EXTERNAL_ID) = UPPER(COALESCE(accounts.ID_1,
accounts.ID_2))
AND accounts.ACCOUNT_ID NOT IN ('TEST_1', 'TEST_2')
```
→ import aborts; nothing is emitted.

## Why this is inconsistent
The importer already preserves MV-only join attributes (cardinality, rely, …) via `write_stash(rel, stash)` into the relationship's `custom_extensions`, and preserves other MV-only features (filter, window, format) the same way. The spec supports `custom_extensions` on relationships. Aborting on an unrepresentable `on` — rather than stashing it like everything else — is the odd one out.

## Proposal
When `on` cannot be decomposed into equi-join column pairs, preserve the raw clause in `relationship.custom_extensions[DATABRICKS]` (e.g. `{"on": ""}`) and emit a warning, instead of raising. To avoid misrepresenting a filtered/ non-equi join as a clean equi-join, do NOT populate `from_columns`/`to_columns` in that case (or omit the core relationship and stash the join at dataset/model level). Gate it behind an opt-in (e.g. `on_unsupported="preserve"` / a `--lenient` CLI flag) so the default lossless-or-fail contract is unchanged.

## Impact
Enables `MV → Ossie → MV` round-tripping of production metric views whose joins carry business logic, which the current importer rejects outright.

## Related
Complements (does not conflict with) the stricter relationship-column validation in #307 and #308 — this is opt-in preservation for round-trip fidelity, not a relaxation of the default validation.

Contributor guide

Open the contributing guide

Research direction

Trace convert_metric_view_to_ossie through _decompose_on() and _convert_join(), then inspect how write_stash() stores relationship custom_extensions. Verify the existing conversion tests and define coverage for the opt-in path: unsupported clauses are preserved with a warning without column mappings, while the default behavior still fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.