confluentinc / confluentinc/dbt-confluent
Tableflow: in-place config updates (formats/storage/retention) without --full-refresh
- Dominant language
- Python
- Stars
- 9
- Forks
- 3
- Avg merge
- 1d 59m
- Merged PRs (30d)
- 18
Description
Allow changing an enabled Tableflow config's `formats`, `storage`, `retention_ms`, `data_retention_ms`, or `error_handling` without `--full-refresh`.
Before implementing anything here, we need answers to:
- **Is disable-then-re-enable actually safe?** Disabling tears down the Iceberg/Delta materialization on the destination side. We don't know whether that preserves the destination table's history, whether re-enabling resumes cleanly, or whether downstream consumers of that Iceberg/Delta table see a discontinuity.
- **Is it cheap enough to do automatically?** `enable_tableflow`/`disable_tableflow` both default to blocking until confirmed (up to 300s each). Reconciling on every run (mirroring how `materialized_table` re-asserts `with`/`statement_properties`) could be expensive if a mismatch is common, and we don't have evidence either way.
This may end up requiring a feature request against `confluent-sql` (a real update/patch API) rather than something safe to fake client-side via disable+enable. Scope this issue as investigation and design; implementation should be a follow-up once the above is answered.
Contributor guide
Research direction
Start by tracing the enable_tableflow and disable_tableflow entry points and compare their blocking behavior with materialized_table reconciliation. Investigate whether disable-then-re-enable preserves destination history and downstream continuity, and whether confluent-sql provides an update or patch API. Done means a documented safety and cost assessment plus a clear design and follow-up implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100