confluentinc / confluentinc/dbt-confluent

Tableflow: in-place config updates (formats/storage/retention) without --full-refresh

Open
#101 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
3
Avg merge
1d 59m
Merged PRs (30d)
18

Description

Allow changing an enabled Tableflow config's `formats`, `storage`, `retention_ms`, `data_retention_ms`, or `error_handling` without `--full-refresh`.

Before implementing anything here, we need answers to:

- **Is disable-then-re-enable actually safe?** Disabling tears down the Iceberg/Delta materialization on the destination side. We don't know whether that preserves the destination table's history, whether re-enabling resumes cleanly, or whether downstream consumers of that Iceberg/Delta table see a discontinuity.
- **Is it cheap enough to do automatically?** `enable_tableflow`/`disable_tableflow` both default to blocking until confirmed (up to 300s each). Reconciling on every run (mirroring how `materialized_table` re-asserts `with`/`statement_properties`) could be expensive if a mismatch is common, and we don't have evidence either way.

This may end up requiring a feature request against `confluent-sql` (a real update/patch API) rather than something safe to fake client-side via disable+enable. Scope this issue as investigation and design; implementation should be a follow-up once the above is answered.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the enable_tableflow and disable_tableflow entry points and compare their blocking behavior with materialized_table reconciliation. Investigate whether disable-then-re-enable preserves destination history and downstream continuity, and whether confluent-sql provides an update or patch API. Done means a documented safety and cost assessment plus a clear design and follow-up implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.