ClickHouse / ClickHouse/dbt-clickhouse

Allow partitioning of the queries needed to build table materializations

Open
#605 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
176
Avg merge
2d 10h
Merged PRs (30d)
8

Description

**Is your feature request related to a problem? Please describe.**
We have found some situations where the query `INSERT INTO x SELECT...` used to build tables was so large that it caused problems on the replica where it was being executed, particularly in terms of memory usage.

**Describe the solution you'd like**
Some users have already implemented their own way of managing this by designating a column to create "partitions," allowing the query to be loaded in smaller chunks.

**Describe alternatives you've considered**
Something like this can already be done by using [microbatches](https://docs.getdbt.com/docs/build/incremental-microbatch), but some users may not want to use this approach.

**Additional context**
Creating this issue to track how users are handling this and the implications, to see if it would make sense to implement it directly in the adapter.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the adapter's current table-materialization path and how it executes the INSERT INTO ... SELECT query. Compare that behavior with the linked microbatch approach and document the implications of partitioning by a designated column. Done should include an agreed implementation scope and validation approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.