dbt-labs / dbt-labs/dbt

Databricks MV/ST get_relation_config does not wait for DLT pipeline IDLE

Open
#16,121 0 comments 0 reactions 0 assignees View on GitHub
adapter:databricks area:adapters
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

## Summary

Fusion's Databricks adapter does not wait for Delta Live Tables (DLT) pipeline completion when loading remote relation config for **materialized views** and **streaming tables**. dbt-databricks does.

**Severity:** Medium (P2) for typical production cadences; higher for CI / back-to-back reruns. Not a first-run create bug and not silent corruption on every run.

## Background / parity

In dbt-databricks, `DeltaLiveTableAPIBase.get_from_relation` describes the relation, then if `tblproperties.pipeline_id` (`pipelines.pipelineId`) is set, calls `api_client.dlt_pipelines.poll_for_completion` until the pipeline is `IDLE` (SDK `wait_get_pipeline_idle`, ~1h timeout). Related history: [databricks/dbt-databricks#674](https://github.com/databricks/dbt-databricks/issues/674).

Fusion already parses `pipelines.pipelineId` out of tblproperties, but never waits. There is an explicit TODO:

- `crates/dbt-adapter/src/metadata/databricks/mod.rs` in `fetch_relation_config_from_remote` — *"TODO: Implement polling for DLT pipeline status / we don't have the dbx client here"*

`REFRESH MATERIALIZED VIEW` / `REFRESH STREAMING TABLE` remain fire-and-forget in both engines. The settle barrier is on the **next** `get_relation_config` (start of a subsequent run's config-change path), not after the current run's REFRESH.

## What is affected today

`get_relation_config` runs for **existing** MV/ST when macros call `get_configuration_changes` to choose REFRESH vs ALTER (`materialized_view.sql` / `streaming_table.sql`). Create and full-refresh skip this path.

If a prior refresh's pipeline is still running when the next run describes the relation:

- Spurious or **missed** configuration diffs (tags, refresh schedule, tblproperties, row filter) → extra ALTER/replace or skipped needed alters
- ALTER / tag apply while a DLT update is in flight
- Flaky tests that assert row counts / config immediately after refresh without an explicit settle
- Failed pipelines are not surfaced as a clear adapter error on this path (users usually see a later SQL error)

**Unaffected:** tables, views, metric views; first create; runs where the previous refresh already reached IDLE; spaced-out daily/hourly jobs that finish before the next invoke (often look healthy).

## Proposed acceptance criteria (when scheduled)

- [ ] After loading MV/ST remote config, if `pipelines.pipelineId` is present, wait until pipeline state is `IDLE` (parity with dbt-databricks timeout / failure semantics)
- [ ] Works for the same auth methods SQL already supports (not PAT-only)
- [ ] Table/View/MetricView and missing pipeline ID do not wait
- [ ] Cancellation does not leave a long blocking poll
- [ ] Replay / recorded `get_relation_config` path does not call the pipeline API
- [ ] Focused unit tests for IDLE / nonterminal→IDLE / FAILED / timeout / cancel / auth skip

## Out of scope for this issue

Implementing the waiter in the current tag-fetch / relation-config skip workstream; this is intentionally deferred.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.