ClickHouse / ClickHouse/dbt-clickhouse

Strategy for breaking down the cost of joins

Open
#142 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

I had a discussion with [Roy Rozenblum](https://clickhousedb.slack.com/archives/D04873AA1A6) on the public Slack and he recommends an interesting approach to spread the cost of joins:

The idea is this: we need to split the RIGHT table into buckets. Assume TabA join TabB.
The formula for creating the buckets is pretty simple:
`toInt64( /* maybe we should hash if key is big */) % ` as bucket.

Then we just loop the INSERT command `` times, with the bucket number as parameter.

There are some challenges for clickhouse:
We can't just add a WHERE condition of `bucket = X` because clickhouse doesn't do smart pushdown yet. So we need to add a subselect with the bucket expression and WHERE clause to wrap around the RIGHT table name / model.
Also we need to make sure it's an INNER JOIN only (it won't work otherwise).
So I think the approach is to add a special JOIN Macro, so instead of writing the join in a regular way, the TabA join TabB on X=Y we put a Macro there the does all of the above.
So something like this (slightly simplified):
/* Assume TabB's primary key is B1, B2 */
```
select * from
{{ inner_join_with_buckets(TabA, TabB, no_of_buckets=16) }} on TabA.A2 = TabB.B3 and TabB.A3 = TabB.B4
where TabA.A1 = 'Whatever'
Would result in this:
select * from
TabA join
(select *, toInt64(someHash(Tuple(B1, B2))) % 16 as this_rows_bucket from TabB where this_rows_bucket = ) as TabB
on TabA.A2 = TabB.B3 and TabB.A3 = TabB.B4
where TabA.A1 = 'Whatever'
```

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating how dbt-clickhouse represents and compiles macros and join SQL, then assess where the proposed bucketed INNER JOIN could fit. Done would require an agreed design, defined implementation scope, and tests for the generated SQL.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.