confluentinc / confluentinc/dbt-confluent

SQL Logic changes silently skip for table and streaming_table

Open
#97 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
2
Avg merge
1d 53m
Merged PRs (30d)
17

Description

...there may be some analog for streaming_source as well, though there is no SQL.

The behavior is documented in MATERIALIZATIONS.md, but when you're actually running your DBT models there is no indication that the SQL has changed and is being ignored.

Note for logic drift detection:
* For `table` models, the statement will eventually reach a terminal state and may be subject to GC, so at some point the client may not be able to perform a diff
+ Using dbt's client-side state manifest could mitigate this issue
* For a `streaming_table`, it's much more likely to be a long-lived statement that we can fetch and perform - at the very least - some naive equality comparison against.

If we can't reliably do the diff, we should consider always pushing the explicit decision to run to the user (e.g. always re-create when the model is run, and point them to dbt's client-side state manifests for diffing)

Contributor guide

Open the contributing guide

Research direction

Start with MATERIALIZATIONS.md and trace the table and streaming_table behavior described there while reviewing how the dbt models are run; determine whether streaming_source is also in scope. Done means SQL changes no longer disappear silently, through a reliable diff or an explicit run decision, with the chosen behavior documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.