confluentinc / confluentinc/dbt-confluent
streaming_table: INSERT is never re-projected to match schema.yml's declared column order
- Dominant language
- Python
- Stars
- 8
- Forks
- 2
- Avg merge
- 1d 59m
- Merged PRs (30d)
- 18
Description
## Summary
_Observed in code, in passing while working on MTs. Needs validation._
`streaming_table`'s long-running `INSERT INTO {{ target_relation }} {{ sql }}` (macros/materializations/models/streaming_table.sql:57) uses the model's SQL as-authored, with no re-projection to match the column order declared in schema.yml's `columns:` block.
This is the same bug class just fixed for `materialized_table` in PR #84 (which was missing `get_select_subquery`'s reprojection in its `CREATE OR ALTER MATERIALIZED TABLE ... AS SELECT`), except here it's unconditional (not contract-gated) and still unfixed. No existing functional test for `streaming_table` exercises mismatched column order, so this gap isn't caught by CI.
## Proposed fix
Add `{%- set sql = get_select_subquery(sql) -%}` once, at the materialization's top level between the `-ddl` call block and the `main`/`INSERT` call block (it can't go inside either `{% call %}`...`{%- endcall %}` body, since those are separate Jinja scopes and a `set` inside one wouldn't be visible to the other). Unlike the `materialized_table` fix, this shouldn't reuse `render_contract_columns_and_reproject_sql` (helpers.sql) as-is, since that macro is gated on `contract.enforced` and `streaming_table`'s column rendering is unconditional -- reusing it would silently skip reprojection for the common case of a model that doesn't set `contract={'enforced': true}`.
Contributor guide
Research direction
Start at macros/materializations/models/streaming_table.sql:57 and compare the materialized_table change in PR #84, focusing on get_select_subquery and the separate -ddl and main/INSERT call blocks; helpers.sql contains the related contract-gated macro. Add functional coverage for a streaming_table whose model SQL order differs from schema.yml, and confirm CI shows the INSERT matching the declared column order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 83/100