[databricks] MV→Ossie import aborts on non-equi/filtered join conditions instead of preserving them in custom_extensions
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 267
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 24
Description
## Summary
`convert_metric_view_to_ossie` raises `ConversionError` and aborts the entire import when a Metric View join's `on` clause is not a simple equi-join of `alias.column = alias.column` pairs. Real-world metric views routinely use function-wrapped keys and filter predicates in join conditions, so this makes the importer unusable on them — even though the converter already has a custom_extensions mechanism that could preserve such joins losslessly.
## Current behavior
`_decompose_on()` returns the raw `on` string for any clause containing a non-equi operator, a SQL fragment (e.g. `UPPER(...)`, `COALESCE(...)`), or an extra filter predicate (`... AND col NOT IN (...)`). `_convert_join()` then:
if raw_on is not None:
raise ConversionError("Join '...' uses a non-equi or unsupported join
condition ... Cannot import.")
Example (generic Databricks Unity Catalog metric view):
```
joins:
- name: accounts
source: catalog.schema.account
on: UPPER(source.EXTERNAL_ID) = UPPER(COALESCE(accounts.ID_1,
accounts.ID_2))
AND accounts.ACCOUNT_ID NOT IN ('TEST_1', 'TEST_2')
```
→ import aborts; nothing is emitted.
## Why this is inconsistent
The importer already preserves MV-only join attributes (cardinality, rely, …) via `write_stash(rel, stash)` into the relationship's `custom_extensions`, and preserves other MV-only features (filter, window, format) the same way. The spec supports `custom_extensions` on relationships. Aborting on an unrepresentable `on` — rather than stashing it like everything else — is the odd one out.
## Proposal
When `on` cannot be decomposed into equi-join column pairs, preserve the raw clause in `relationship.custom_extensions[DATABRICKS]` (e.g. `{"on": ""}`) and emit a warning, instead of raising. To avoid misrepresenting a filtered/ non-equi join as a clean equi-join, do NOT populate `from_columns`/`to_columns` in that case (or omit the core relationship and stash the join at dataset/model level). Gate it behind an opt-in (e.g. `on_unsupported="preserve"` / a `--lenient` CLI flag) so the default lossless-or-fail contract is unchanged.
## Impact
Enables `MV → Ossie → MV` round-tripping of production metric views whose joins carry business logic, which the current importer rejects outright.
## Related
Complements (does not conflict with) the stricter relationship-column validation in #307 and #308 — this is opt-in preservation for round-trip fidelity, not a relaxation of the default validation.
Contributor guide
Research direction
Trace convert_metric_view_to_ossie through _decompose_on() and _convert_join(), then inspect how write_stash() stores relationship custom_extensions. Verify the existing conversion tests and define coverage for the opt-in path: unsupported clauses are preserved with a warning without column mappings, while the default behavior still fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100