Databricks MV/ST get_relation_config does not wait for DLT pipeline IDLE
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
## Summary
Fusion's Databricks adapter does not wait for Delta Live Tables (DLT) pipeline completion when loading remote relation config for **materialized views** and **streaming tables**. dbt-databricks does.
**Severity:** Medium (P2) for typical production cadences; higher for CI / back-to-back reruns. Not a first-run create bug and not silent corruption on every run.
## Background / parity
In dbt-databricks, `DeltaLiveTableAPIBase.get_from_relation` describes the relation, then if `tblproperties.pipeline_id` (`pipelines.pipelineId`) is set, calls `api_client.dlt_pipelines.poll_for_completion` until the pipeline is `IDLE` (SDK `wait_get_pipeline_idle`, ~1h timeout). Related history: [databricks/dbt-databricks#674](https://github.com/databricks/dbt-databricks/issues/674).
Fusion already parses `pipelines.pipelineId` out of tblproperties, but never waits. There is an explicit TODO:
- `crates/dbt-adapter/src/metadata/databricks/mod.rs` in `fetch_relation_config_from_remote` — *"TODO: Implement polling for DLT pipeline status / we don't have the dbx client here"*
`REFRESH MATERIALIZED VIEW` / `REFRESH STREAMING TABLE` remain fire-and-forget in both engines. The settle barrier is on the **next** `get_relation_config` (start of a subsequent run's config-change path), not after the current run's REFRESH.
## What is affected today
`get_relation_config` runs for **existing** MV/ST when macros call `get_configuration_changes` to choose REFRESH vs ALTER (`materialized_view.sql` / `streaming_table.sql`). Create and full-refresh skip this path.
If a prior refresh's pipeline is still running when the next run describes the relation:
- Spurious or **missed** configuration diffs (tags, refresh schedule, tblproperties, row filter) → extra ALTER/replace or skipped needed alters
- ALTER / tag apply while a DLT update is in flight
- Flaky tests that assert row counts / config immediately after refresh without an explicit settle
- Failed pipelines are not surfaced as a clear adapter error on this path (users usually see a later SQL error)
**Unaffected:** tables, views, metric views; first create; runs where the previous refresh already reached IDLE; spaced-out daily/hourly jobs that finish before the next invoke (often look healthy).
## Proposed acceptance criteria (when scheduled)
- [ ] After loading MV/ST remote config, if `pipelines.pipelineId` is present, wait until pipeline state is `IDLE` (parity with dbt-databricks timeout / failure semantics)
- [ ] Works for the same auth methods SQL already supports (not PAT-only)
- [ ] Table/View/MetricView and missing pipeline ID do not wait
- [ ] Cancellation does not leave a long blocking poll
- [ ] Replay / recorded `get_relation_config` path does not call the pipeline API
- [ ] Focused unit tests for IDLE / nonterminal→IDLE / FAILED / timeout / cancel / auth skip
## Out of scope for this issue
Implementing the waiter in the current tag-fetch / relation-config skip workstream; this is intentionally deferred.
Contributor guide
Assessment
This issue has not been assessed yet.