confluentinc / confluentinc/dbt-confluent
Cascade restart of downstream consumers on stateful model change
- Dominant language
- Python
- Stars
- 8
- Forks
- 2
- Avg merge
- 1d 59m
- Merged PRs (30d)
- 18
Description
When a stateful model (one using JOIN, GROUP BY, OVER) is redeployed, downstream consumers that read from it may need to be restarted to avoid silent data corruption. Currently the adapter does not handle this.
### Open questions
- Does dbt's DAG execution already handle this by re-running downstream models in dependency order?
- Does dropping and recreating a table automatically invalidate downstream statements, or do they silently continue reading from the recreated table?
- Is any special logic needed beyond recreating the downstream table/statement?
These questions need to be investigated before deciding on an implementation approach.
Contributor guide
Research direction
Investigate how dbt's DAG execution handles downstream models when a stateful model using JOIN, GROUP BY, or OVER is redeployed. Check whether dropping and recreating tables or statements invalidates downstream consumers or leaves them reading silently; done means documenting the behavior and identifying whether adapter-specific restart logic is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100