confluentinc / confluentinc/dbt-confluent

Cascade restart of downstream consumers on stateful model change

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
2
Avg merge
1d 59m
Merged PRs (30d)
18

Description

When a stateful model (one using JOIN, GROUP BY, OVER) is redeployed, downstream consumers that read from it may need to be restarted to avoid silent data corruption. Currently the adapter does not handle this.

### Open questions

- Does dbt's DAG execution already handle this by re-running downstream models in dependency order?
- Does dropping and recreating a table automatically invalidate downstream statements, or do they silently continue reading from the recreated table?
- Is any special logic needed beyond recreating the downstream table/statement?

These questions need to be investigated before deciding on an implementation approach.

Contributor guide

Open the contributing guide

Research direction

Investigate how dbt's DAG execution handles downstream models when a stateful model using JOIN, GROUP BY, or OVER is redeployed. Check whether dropping and recreating tables or statements invalidates downstream consumers or leaves them reading silently; done means documenting the behavior and identifying whether adapter-specific restart logic is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.