ClickHouse / ClickHouse/dbt-clickhouse

PREWHERE (and similar native constructs) don't work with unit tests

Open
#551 5 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
176
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug

PREWHERE (and similar native constructs) don't work with unit tests. We leverage PREWHERE in certain performance-critical models where we are certain that we can run FINAL after PREWHERE.

### Steps to reproduce (with pseudo code)

```
#_my_fancy_model_
select *
from {{ source("raw", "my_fancy_source") }}
prewhere id = 5
```

1. Implement a model with a PREWHERE filtering on an arbitrary source table's column.
2. Run a unit test for the model
3. Error:

Received ClickHouse exception, code: 182, server response: Code: 182. DB::Exception: Invalid column __table3.id in PREWHERE. Expected column source to be table or table function. Actual __dbt__cte__my_fancy_model ...

### Expected behaviour

Should be able to perform PREWHERE filtering on the sources.

I believe the issue stems from the fact unit test sources (sources, not the model itself, but sources!) in `given` are inlined as CTEs instead of seeds or temporary tables. Clickhouse disallows to prewhere filter on CTEs.

Any ideas how to approach this? Don't think dbt will release a fix anytime soon, since it's a Clickhouse-specific issue... We're thinking of overriding this (somehow) locally, but maybe we could implement something within the level of this adapter.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the unit test with a source CTE and PREWHERE, then inspect how sources supplied in `given` are inlined during unit-test compilation. Compare that generated query with ClickHouse's PREWHERE restriction on CTEs; done means the unit test can apply PREWHERE to the source without the reported exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.