ClickHouse / ClickHouse/dbt-clickhouse
Allow partitioning of the queries needed to build table materializations
- Dominant language
- Python
- Stars
- 362
- Forks
- 176
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
**Is your feature request related to a problem? Please describe.**
We have found some situations where the query `INSERT INTO x SELECT...` used to build tables was so large that it caused problems on the replica where it was being executed, particularly in terms of memory usage.
**Describe the solution you'd like**
Some users have already implemented their own way of managing this by designating a column to create "partitions," allowing the query to be loaded in smaller chunks.
**Describe alternatives you've considered**
Something like this can already be done by using [microbatches](https://docs.getdbt.com/docs/build/incremental-microbatch), but some users may not want to use this approach.
**Additional context**
Creating this issue to track how users are handling this and the implications, to see if it would make sense to implement it directly in the adapter.
Contributor guide
Research direction
Start by reviewing the adapter's current table-materialization path and how it executes the INSERT INTO ... SELECT query. Compare that behavior with the linked microbatch approach and document the implications of partitioning by a designated column. Done should include an agreed implementation scope and validation approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100