confluentinc / confluentinc/dbt-confluent

Schema drift check fails for models with a leading WITH clause (Table 'EXPR$0' not found)

Open
#75 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
8
Forks
2
Avg merge
1d 59m
Merged PRs (30d)
18

Description

## Summary

The schema-drift check fails for any model whose compiled SQL starts with a `WITH` clause — i.e. most real dbt models. The model deploys and re-deploys fine with `--full-refresh`, but a plain re-run errors out:

```
Database Error in model stg_semen_batch (models\staging\stg_semen_batch.sql)
confluent_sql error for '/* {"app": "dbt", "dbt_version": "1.11.12", ...} */

CREATE TABLE `env-xxxxxx`.`my-cluster`.`__dbt_tmp_schema_check_stg_semen_batch` AS
SELECT * FROM (

WITH stg AS (
SELECT
CAST(num AS CHAR(6)) AS num,
-- more columns here
FROM `dev`.`my-cluster`.`some_source`
)
SELECT
stg.num,
-- more columns here
FROM stg
) WHERE FALSE
': Statement submission failed: SQL validation failed. Error at or near line 4, column 14.

Caused by: Table 'EXPR$0' not found
```

Reported by a user; reproduced conceptually against the current `main` (0.3.0).

## Root cause

`check_for_schema_drift` builds the drift-check temp table by wrapping the model SQL in a derived table so it can append a row-suppressing filter — `dbt/include/confluent/macros/materializations/models/helpers.sql:192-198`:

```sql
CREATE TABLE {{ temp_relation }} AS
SELECT * FROM (
{{ sql }}
) WHERE FALSE
```

When `{{ sql }}` begins with a CTE, this puts a `WITH` query in a `FROM`-derived-table position:

```sql
SELECT * FROM ( WITH stg AS (…) SELECT … FROM stg ) WHERE FALSE
```

`EXPR$0` is Calcite's generated alias for an unaliased `FROM` item. `Table 'EXPR$0' not found` at the outer `SELECT *` is the validator failing to resolve that synthetic alias, because the subquery node is a `SqlWith` rather than a plain `SELECT`.

The same model SQL validates fine through the normal build path, which never wraps it — `confluent__create_table_as` emits `create table … as ( {{ sql }} )` (`dbt/include/confluent/macros/relations/table/create.sql:17-19`).

That asymmetry is exactly why `--full-refresh` masks the bug:

- `--full-refresh` → `decide_action` returns `create` → table dropped and rebuilt via `confluent__create_table_as`. No drift check.
- plain run → relation exists → `_prepare_skip` → `_enforce_schema_drift` with the default `on_schema_drift: 'fail'` (`helpers.sql:124-140`) → drift check → failing statement above.

## Suggested fix

Stop putting the model SQL in a `FROM` clause at all. Build the drift-check temp table with the same parenthesized CTAS form that is already proven to work, and get the "no rows" effect by appending a row limit instead of wrapping:

```sql
CREATE TABLE {{ temp_relation }} AS (
{{ sql }}
)
LIMIT 0
```

This keeps any leading `WITH` at the top level of the statement, where Flink validates it correctly, and leaves the `streaming_source` branch (`helpers.sql:199-204`, column definitions) untouched.

Points to verify before/while implementing:

1. **`LIMIT` in streaming mode.** Confluent Cloud documents the [`LIMIT` clause](https://docs.confluent.io/cloud/current/flink/reference/queries/limit.html) but does not state whether it is accepted in streaming mode. Needs a live check against Confluent Cloud.
2. **Statement must still terminate without processing data.** `WHERE FALSE` currently lets the planner prune the source so the statement completes immediately. `LIMIT 0` must give the same property — a temp-table statement that instead starts a long-running job reading from the source would be a regression in cost and runtime. If it does not, alternatives to evaluate: keeping the derived-table wrapper but adding an explicit alias (`SELECT * FROM ( {{ sql }} ) AS dbt_schema_check WHERE FALSE`), or deriving the expected schema without a temp table at all.
3. **Appending to arbitrary compiled SQL.** A model that already ends in `LIMIT n`, or in a trailing line comment, would break a naive concatenation. Worth confirming the compiled SQL dbt hands the materialization is safe to append to, or normalizing it first.

## Test coverage gap

Nothing under `tests/` exercises a model whose SQL contains a CTE, which is why this path shipped broken. The fix should add a functional test that runs a CTE-based model twice (second run hitting the drift check) for both `table` and `streaming_table`.

## Workaround for affected users

Set `on_schema_drift: ignore` on the model (or project-wide). For `table` materializations the skip path passes `restart=false`, so `_enforce_schema_drift` runs no check at all (`helpers.sql:141-144`) and the re-run is a clean no-op. Use `--full-refresh` for real schema changes.

Contributor guide

Open the contributing guide

Research direction

Start in dbt/include/confluent/macros/materializations/models/helpers.sql:192-204 and compare the drift-check CTAS with confluent__create_table_as in dbt/include/confluent/macros/relations/table/create.sql:17-19. Reproduce a CTE-based model twice, then verify the chosen fix against Confluent Cloud, including streaming behavior and immediate completion. Add functional coverage under tests/ for both table and streaming_table materializations; done means the second run reaches schema-drift checking without validation errors or an unintended long-running job.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.