ClickHouse / ClickHouse/dbt-clickhouse
Strategy for breaking down the cost of joins
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
I had a discussion with [Roy Rozenblum](https://clickhousedb.slack.com/archives/D04873AA1A6) on the public Slack and he recommends an interesting approach to spread the cost of joins:
The idea is this: we need to split the RIGHT table into buckets. Assume TabA join TabB.
The formula for creating the buckets is pretty simple:
`toInt64( /* maybe we should hash if key is big */) % ` as bucket.
Then we just loop the INSERT command `` times, with the bucket number as parameter.
There are some challenges for clickhouse:
We can't just add a WHERE condition of `bucket = X` because clickhouse doesn't do smart pushdown yet. So we need to add a subselect with the bucket expression and WHERE clause to wrap around the RIGHT table name / model.
Also we need to make sure it's an INNER JOIN only (it won't work otherwise).
So I think the approach is to add a special JOIN Macro, so instead of writing the join in a regular way, the TabA join TabB on X=Y we put a Macro there the does all of the above.
So something like this (slightly simplified):
/* Assume TabB's primary key is B1, B2 */
```
select * from
{{ inner_join_with_buckets(TabA, TabB, no_of_buckets=16) }} on TabA.A2 = TabB.B3 and TabB.A3 = TabB.B4
where TabA.A1 = 'Whatever'
Would result in this:
select * from
TabA join
(select *, toInt64(someHash(Tuple(B1, B2))) % 16 as this_rows_bucket from TabB where this_rows_bucket = ) as TabB
on TabA.A2 = TabB.B3 and TabB.A3 = TabB.B4
where TabA.A1 = 'Whatever'
```
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by locating how dbt-clickhouse represents and compiles macros and join SQL, then assess where the proposed bucketed INNER JOIN could fit. Done would require an agreed design, defined implementation scope, and tests for the generated SQL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100