confluentinc / confluentinc/dbt-confluent

streaming_table: INSERT is never re-projected to match schema.yml's declared column order

Open Beginner friendly
#89 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
2
Avg merge
1d 59m
Merged PRs (30d)
18

Description

## Summary

_Observed in code, in passing while working on MTs. Needs validation._

`streaming_table`'s long-running `INSERT INTO {{ target_relation }} {{ sql }}` (macros/materializations/models/streaming_table.sql:57) uses the model's SQL as-authored, with no re-projection to match the column order declared in schema.yml's `columns:` block.

This is the same bug class just fixed for `materialized_table` in PR #84 (which was missing `get_select_subquery`'s reprojection in its `CREATE OR ALTER MATERIALIZED TABLE ... AS SELECT`), except here it's unconditional (not contract-gated) and still unfixed. No existing functional test for `streaming_table` exercises mismatched column order, so this gap isn't caught by CI.

## Proposed fix

Add `{%- set sql = get_select_subquery(sql) -%}` once, at the materialization's top level between the `-ddl` call block and the `main`/`INSERT` call block (it can't go inside either `{% call %}`...`{%- endcall %}` body, since those are separate Jinja scopes and a `set` inside one wouldn't be visible to the other). Unlike the `materialized_table` fix, this shouldn't reuse `render_contract_columns_and_reproject_sql` (helpers.sql) as-is, since that macro is gated on `contract.enforced` and `streaming_table`'s column rendering is unconditional -- reusing it would silently skip reprojection for the common case of a model that doesn't set `contract={'enforced': true}`.

Contributor guide

Open the contributing guide

Research direction

Start at macros/materializations/models/streaming_table.sql:57 and compare the materialized_table change in PR #84, focusing on get_select_subquery and the separate -ddl and main/INSERT call blocks; helpers.sql contains the related contract-gated macro. Add functional coverage for a streaming_table whose model SQL order differs from schema.yml, and confirm CI shows the INSERT matching the declared column order.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
83/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.