ClickHouse / ClickHouse/dbt-clickhouse
PREWHERE (and similar native constructs) don't work with unit tests
- Dominant language
- Python
- Stars
- 362
- Forks
- 176
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
PREWHERE (and similar native constructs) don't work with unit tests. We leverage PREWHERE in certain performance-critical models where we are certain that we can run FINAL after PREWHERE.
### Steps to reproduce (with pseudo code)
```
#_my_fancy_model_
select *
from {{ source("raw", "my_fancy_source") }}
prewhere id = 5
```
1. Implement a model with a PREWHERE filtering on an arbitrary source table's column.
2. Run a unit test for the model
3. Error:
Received ClickHouse exception, code: 182, server response: Code: 182. DB::Exception: Invalid column __table3.id in PREWHERE. Expected column source to be table or table function. Actual __dbt__cte__my_fancy_model ...
### Expected behaviour
Should be able to perform PREWHERE filtering on the sources.
I believe the issue stems from the fact unit test sources (sources, not the model itself, but sources!) in `given` are inlined as CTEs instead of seeds or temporary tables. Clickhouse disallows to prewhere filter on CTEs.
Any ideas how to approach this? Don't think dbt will release a fix anytime soon, since it's a Clickhouse-specific issue... We're thinking of overriding this (somehow) locally, but maybe we could implement something within the level of this adapter.
Contributor guide
Research direction
Start by reproducing the unit test with a source CTE and PREWHERE, then inspect how sources supplied in `given` are inlined during unit-test compilation. Compare that generated query with ClickHouse's PREWHERE restriction on CTEs; done means the unit test can apply PREWHERE to the source without the reported exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100